Identifying hierarchical choice structures: a comparison of methods.

ABSTRACT

Consumers make product choice decisions based on a personal evaluation of the product's attributes. The hierarchical theory posits that a choice is first made on the most important attribute dimension, followed by the next most important, etc, until a unique product is chosen. This paper compares three methods for identifying the hierarchical structure based on product switching matrices. To use multidimensional scaling and hierarchical cluster analysis the paper develops a dissimilarity measure from the switching matrix. The third method uses entropy to unfold the hierarchy directly from the switching matrix. A numerical example compares the relative effectiveness of the three methods.

Keywords: Market Research, Market Partitioning, Entropy, Cluster Analysis, Multidimensional Scaling

1. INTRODUCTION

The problem of market partitioning is to find a way to group products into subgroups that directly compete with each other. Products can be considered to compete when potential purchasers perceive them as satisfying the same requirements. An example of the market for table spreads might include all products that are spread on bread, including cheeses, butter, margarine and margarine substitutes. Each of these product classifications may be seen as a partition of the market for table spreads. Products within a partition, such as margarine, would be expected to compete more closely with other margarine products than with products in other partitions, such as cheeses. Within the margarine subset, products can be further partitioned into competing subsets by other attribute dimensions such as brand, form, saltiness, consistency, oil type, etc.

Product design, advertising, promotion and pricing are examples of strategic decisions that are based on how a market is partitioned. In the margarine market, the total calories in a serving might determine the competing subset for some potential purchasers. Reducing the number of calories from the low 90's to the high 70's might allow the product to be considered by a large number of new consumers looking for a diet alternative. Similarly, if a firm knows which product variants are most closely competing with its own variant, it will be able to more effectively choose advertising themes and particular product attributes to emphasize in advertisements.

Product attributes may be subjective or objective. Brand name itself is an important attribute dimension that has a variety of connotations to customers or potential customers. General Motors sells Chevrolet, Pontiac and Buick models that differ little except by brand, yet potential purchasers do not view them as closely competing products. The subjective associations that go with brand name may well result in their being perceived to be in different market partitions. Products only belong in a competitive subset if potential customers see them as competitive. It is therefore insufficient to assemble a list of easily measurable attributes for a set of products and then group product variants on the basis of attribute commonality. Because consumers consider both objective and subjective attributes, it is not sufficient to group products by commonality of attribute dimensions. Product utility for any given attribute for one consumer may differ from the utility derived by that attribute for a different consumer. Furthermore, the same consumer may value the attributes of a given product variant differently on another purchase occasion.

The purpose of market partitioning is to define market in terms of a hierarchical structure of attributes. The starting point is to define all product alternatives as a vector of attributes, using all relevant objective and subjective attribute dimensions. For example, margarines could be defined by the attributes: brand, consistency, type of oil, form, size, and degree of "dietness." One product might be the vector (Mazola, solid, corn oil, bar, 16 oz., non-diet.)

Attribute dimensions exhibiting decreasing degrees of customer loyalty determine the attribute hierarchy. The attribute dimension exhibiting the greatest degree of consumer loyalty occupies the top level of the hierarchy. Product variants in top-level partitions are most closely competing, in the sense that there is strong resistance to switching from one partition to another. For the margarine example, if oil type were at the top of the hierarchy, the Mazola product described above might not have to compete on the basis of oil type. Consumers would more readily substitute different brands, sizes, or even "dietness" than they would different oil types. The second level in the hierarchy is the attribute dimension that exhibits the next highest loyalty within the primary partitions. This analysis continues for lower level partitions.

To illustrate the attribute hierarchical model of a market, consider a market with five product variants defined on three attribute dimensions: brand, color and size. Assume that there are three brands (a, b, c), two colors (x, y) and four sizes (r, s, t, u). Further assume that the five product variants are as follows:

An attribute hierarchy with the order: size, color, and brand partitions the market into four competing subsets by size. Consumers in that market switch relatively little among sizes. Because of the particular colors available within sizes, there is no opportunity for consumers to switch among different colors unless they are prepared to change to a different package size. At the lowest level in the hierarchy, consumers have the least resistance to switching. If one brand is not available, another will do almost as well or as well. The attribute hierarchy may be shown as a graph.

2. SOURCES OF DATA

In order to empirically identify how a set of product variants is partitioned, data reflecting the relationship among the variants is needed. We have already argued that objective similarity may not correlate well with perceived substitutability. A common way of trying to measure perceived relationships among attributes is to ask a structured set of questions of a sample of consumers. Aside from the high cost of obtaining such data for a large sample of individuals, one weakness of this approach is that respondents are placed in a hypothetical situation and reported relationship data may well be at odds with relationships revealed by actual purchase behavior. The extent to which attributes are substitutable is more reliably measured by recording the attributes actually chosen to meet a particular need. For example, if a person is observed to eat at a coffee shop, then a fast-food restaurant, followed by a steak house, it would be fallacious to conclude that the three types of restaurant are closely competitive or close substitutes. Ideally then, context should be controlled for when collecting data on substitutability patterns. The methodologies compared in this paper are relevant for frequently purchased consumer products. They are not suitable for expensive, durable goods since the purchaser's utility function probably changes from one purchase occasion to the next. Diary panel data, though imperfect, are readily obtainable and usually record use context.

2.1 Sample Data

Continuing with the example in the section 1, assume that a diary panel recording 100 sequential purchases of a product produced the switching matrix in Table 2. For example, for 17 transaction pairs beginning with the purchase of variant A, 10 transactions were repeat purchases of A, while 1 shows a switch to B, 1 to D and 5 to E.

3. DEVELOPING A DISSIMILARITY MEASURE

The first two methods used to identify the hierarchical structure of a market are multidimensional scaling and hierarchical clustering. Both methods begin with a matrix of "distances" between product pairs. A similarity index is first calculated for each pair of products by finding the ratio of the proportion of consumers switching from i to j to the expected proportion switching from i to j. The expected proportion switching from i to j assumes independence of choice on two successive purchase occasions.

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where: [S.sub.i,j] = the similarity index,

[n.sub.i,j] = the number of switches from product i to product j,

[n.sub.i] = the number of purchases of item i on the first choice occasion,

[n.sub.j] = the number of purchases of item j on the second choice occasion,

n = the total number of purchases.

The distance measure between products i and j is given by:

[D.sub.i,j] = max([S.sub.i,j]) - [S.sub.i,j]

Because the distance measures may not be symmetrical, i.e., [X.sub.1,2] [not equal to] [X.sub.2,1], an average distance measure is found by the following:

[D'.sub.i,j] = [D.sub.i,j] + [D.sub.j,i]/2

This gives us the following dissimilarity matrix for our example:

TABLE 3: DISSIMILARITY MATRIX

        A         B         C         D         E

A     1.526     4.102     4.467     4.363     3.607
B     4.102     0.000     4.458     4.184     4.385
C     4.467     4.458     1.681     3.160     4.206
D     4.363     4.184     3.160     1.808     4.541
E     3.607     4.385     4.206     4.541     2.109

These dissimilarities are ratio scaled, with larger numbers reflecting greater dissimilarities between products based on switching behavior.

4. ATTRIBUTE HIERARCHY METHODS

Methods for identifying an attribute hierarchy may be classified as either empirical or based on a model of market behavior. In the latter case, exemplified by the Hendry system, the market is assumed to be in equilibrium and expected switching frequencies computed for a candidate hierarchical structure. If the empirical data are consistent with a particular proposed hierarchy, the problem has been solved. If not, another structure is proposed until the structure of best fit is obtained.

The present study compares three empirical approaches to uncovering the attribute hierarchy. Multidimensional scaling and hierarchical cluster analysis are two traditional multivariate statistical methods. The third method is based on the information theoretic concept of entropy.

4.1 Multidimensional Scaling

Multidimensional scaling creates a perceptual map, which is a spatial description of a respondent's perception about a product, service or other object of interest. With multidimensional scaling, items that are perceived to be similar will fall close together in multidimensional space and items that are perceived to be dissimilar will be further apart. In the present context, the purpose is to identify two or more dimensions that show the positioning of each product in relation to the others. Based on how the products are positioned on each of the derived dimensions, we may be able to give meaningful labels to the dimensions. For example, if the products were to appear as two distinct groups {C, D} and {A, B, E} along the horizontal axis, we might deduce that the product set was partitioned by the color attribute, knowing that the first group has color x and the second color y. The question is whether, given switching proximities between all product pairs, we can uncover the underlying hierarchical structure of attribute dimensions for the set of products.

In this study, the ALSCAL multidimensional scaling module in the SPSS program was applied to the proximity matrix. The algorithm seeks the spatial pattern that best reproduces the original order of the data. Referring to the proximity matrix, table 3, products C and D have the smallest dissimilarity score and should appear closest together in multidimensional space. By contrast, D and E are most dissimilar and should be farthest apart.

The objective is to use the fewest dimensions to provide a good fit between the original data and the distances in the derived multidimensional space. Kruskars stress index is computed to indicate the goodness of fit, with 0 indicating a perfect fit and 1 indicating the worst fit. In effect, the badness-of-fit is measured as the square root of the differences between the optimally transformed data and the squared distances, as a proportion of the total sum of squares. Similarly, R-SQ measures the proportion of variance of the scaled data (disparities) in the partition, which is accounted for by their corresponding distances.

Analysis of the proximity matrix in the ALSCAL multidimensional scaling routine of SPSS produced a Kruskal's stress index of 0.1971 and an R-SQ of 0.9380 for the matrix. The relatively high stress index indicates that the derived distances in 2-dimensional space provide a poor fit to the original proximities. Given the proximity matrix with ten distances, this is the maximal dimensionality obtainable by ALSCAL. The resulting configuration is shown in figure 1.

The grouping on dimension 1 in the plot in figure 2 indicates that products C and D are similar to one another and dissimilar from products A, E and B. Referring to the product descriptions in table 1, color is the only attribute products C and D have in common. Products A, E and B all have a different color. This suggests that dimension 1 might be interpreted as a color dimension. The grouping on dimension 2 is less clear. Projecting the points onto the vertical axis reveals three groups: {C, E}, {A, D} and on its own, {B}. Again referring to table 1, neither the first nor second subgroup has any attribute in common. Thus, it is not possible to give a meaningful interpretation to dimension 2. It is therefore evident that at best, the color attribute is the only one to be associated with the multidimensional configuration.

[FIGURE 2 OMITTED]

4.2 Hierarchical Cluster Analysis

Hierarchical cluster analysis is a traditional statistical method used for grouping objects into clusters in such a way that a particular object within one cluster is more similar to objects within that cluster than to objects in other clusters. In this way we can see which products are most similar to other products. Objects are grouped based on their similarities on predetermined attributes.

The agglomerative hierarchical cluster method starts with a separate cluster for each product and combines the closest products into a cluster. Then the next closest product is either added to the first cluster or starts a separate cluster. At each successive step, a product is added to an existing cluster or forms another separate cluster. Clusters can be combined but can never be split apart.

When a cluster consists of more than one point, a measure of closeness between clusters must be chosen. The nearest neighbor or single linkage method is one common method. The closest point in one cluster is measured to the closest point in a second cluster to get the Euclidean distance. This is shown in figure 3, below. This method only uses one point in each cluster to define the distance between clusters. The average linkage between clusters uses information about all pairs of distances. The distance between each point in one cluster to each point in the second cluster is calculated and averaged to get the between cluster distance. This method is preferred over the single point method and is the one used for this analysis.

[FIGURE 3 OMITTED]

In our problem, we wish to find a product attribute hierarchy based on actual customer behavior instead of on a predetermined list of product attributes. In addition, attributes can be non-metric such as color, taste, smell, etc. In order to accommodate these differences, we base our proximity matrix on product switching data from consumer panel data (see table 2.) These data show the frequency of transitions from one product to another (including the same product) made by consumers. As discussed in the previous section, the first variant starts as an individual cluster. Each new variant is then added to an existing cluster or forms a new cluster. As each new variant is added, clusters are created and grow in size. As necessary, clusters can be combined to form one larger cluster. The process continues until there is one large cluster. The results are shown in an agglomeration schedule and on a dendrogram. Using the above proximity matrix in the SPSS hierarchical cluster analysis module produced the following results:

FIGURE 4: AGGLOMERATION SCHEDULE

Agglomeration Schedule

              Cluster Combined

Stage     Cluster 1     Cluster 2     Coefficients

1             3             4            3.160
2             1             5            3.607
3             1             2            4.244
4             1             3            4.370

            Stage Cluster Firs
                Appears

Stage     Cluster 1     Cluster 2      Next Stage

1             0             0              4
2             0             0              3
3             2             0              4
4             3             1              0

The agglomeration schedule depicted in figure 4 shows how the clusters were formed and the values of the average distance between clusters. When the coefficient is small, fairly similar clusters are being merged. Large coefficients indicate that dissimilar clusters are being merged. In our example, merging points 3 and 4 resulted in a small coefficient so the new cluster is fairly homogeneous. The result is similar for points 1 and 5. However, when point 2 is merged with the cluster containing points 1 and 5, the coefficient rises, indicating that the newly formed cluster is less homogeneous. This would be a reasonable point to stop merging points and clusters.

The dendrogram in figure 5 is a pictorial representation of the agglomeration schedule. It is scaled so that the coefficients represent values from 0 to 25. As can be seen, clusters {3, 4} and {1, 5} were formed relatively early in the process. Point 2 was added near the end and the last cluster to form a combination of all clusters. A reasonable place to stop merging points or clusters would be after the first two clusters were formed leaving us with clusters made up of {3, 4} and {1, 5} with point {2} as a separate cluster. In our example points 1 through 5 represent products A through E. We are left with the three clusters {C, D}, {A, E} and {B}. Figure 6 depicts the product clusters with their associated attributes. Cluster {C, D} has no switching on brand. Also, cluster {A, E} has no switching on brand. Therefore, brand is selected as being the highest attribute. Since cluster {C, D} also has no switching on color, it is selected as the second highest attribute. Size would be the lowest attribute.

[FIGURE 5 OMITTED]

The attribute hierarchy is pictured in figure 7.

[FIGURE 7 OMITTED]

4.3 Entropy-based Partitioning

Shannon's definition of the entropy H{[p.sub.i]} of a stochastic system with n states, having probability [p.sub.i] is given by:

H{[p.sub.i]} = -k [n.summation over (i=1)] [p.sub.i] log [p.sub.i]

where k is an arbitrary constant. Choosing k = 1, the function has its maximum value of log n when the n states are equally likely. The entropy theory holds that all systems will be at the maximum entropy consistent with the constraints on the system. One of the strengths of this measure is that it can be used to study dependencies in sequences of outcomes. First, second and third order dependencies may be analyzed quite readily. In the present context, a first order process is assumed, where the last item purchased depends only on the one purchased the time before last. By dividing switching frequencies in a row by the total row frequency, row-conditional relative frequencies may be computed. These will be treated as conditional probabilities. From these conditional probabilities and market share probabilities, a measure of switching entropy can be computed.

[H.sub.s] = [summation over (i)][p.sub.i] [summation over (j)] [p.sub.j\i] log [p.sub.j\i]

where [p.sub.i] = state i's share of the market and [p.sub.j|i] = the conditional probability of choosing j next time given that i was chosen last time

The assumption underlying the attribute hierarchy is that consumers view product variants in a competitive subgroup as relatively close substitutes. A high level attribute exhibits relatively low switching, and therefore switching entropy, across states (for example, from one size to another.) Similarly, low level attributes exhibit relatively high switching entropy.

Level 1

* The first step is to compute attribute switching matrices, which are as follows:

FIGURE 8: ATTRIBUTE SWITCHING MATRICES

                    BRAND

                      TO

               A      B      C

FROM     A     39      2      4
         B      3     11      3
         C      4      1     33

                 COLOR

                  TO

               X      Y

FROM     X     46      9
         Y     13     32

                        SIZE

                         TO

               R      S      T      U

FROM     R     25      2      6      5
         S      4     11      1      1
         T      6      0      9      2
         U      6      1      2     19

* The next step is to compute the switching entropy for each attribute dimension. The switching entropy value for Brand is 0.490, for Size it is 0.688 and for Color it is 0.744. For example, the calculations for Color are as follows:

FIGURE 9: ENTROPY CALCULATION EXAMPLE

Example: Color

                             TO

                     Color x     Color y     [P.sub.i]

FROM     Color x      0.836       0.164        0.55
         Color y      0.289       0.711        0.45

[H.sub.s] = 0.516

Adjusted [H.sub.s] = [H.sub.s]/log(2): 0.744

Notes: (1.) All logs are natural logs.

(2.) The adjustment is for the number of states

* At the third step, the attribute dimension with the lowest entropy is selected as the highest level attribute in the hierarchy. In the example, Brand is at the top level.

Level 2

* Since this is a brand primary market, only transaction pairs with common brand names are considered in computing brand-conditional switching matrices for Color and Size within each brand.

* A weighted average switching entropy is computed using as weights the market shares of the brands. The conditional switching entropy values for the two second-level candidates is 0.363 for Color and 0.712 for Size, thus indicating a Brand--Color--Size hierarchy.

[FIGURE 10 OMITTED]

This attribute hierarchy indicates that consumers show relatively little loyalty to size variants, so that A and E are strongly competitive, as are D and C. Consumers are less likely to switch to B from the other variants, so is less competitive and more immune to marketing efforts to attract consumers of B. Note that for this example, the entropy-based method gave the same attribute hierarchy structure as the hierarchical cluster analysis shown in figure 4.

5. SUMMARY AND CONCLUSIONS

This paper has compared three empirical methods for identifying competing sub-groups of products from the analysis of product switching matrices. All methods require the pre-specification of product variants as vectors of their attributes. Hierarchical cluster analysis is normally applied to data recording attribute values on metric scales, while multidimensional scaling can proceed from either metric or non-metric similarity or dissimilarity data. For metric data, it is simple to construct a distance measure (often Euclidean) to compute a semi-matrix of distances to serve as the input to these algorithms. However, many product attributes of interest to marketers are non-metric and it is less clear how to construct distances among product alternatives. In this paper we demonstrated how to construct a distance measure between pairs of product variants from a non-symmetrical transition matrix showing frequency of purchases of one variant following purchases of each other variant.

Multidimensional scaling was able to reveal little about the attribute structure. Although the configuration plot in two dimensions suggested that color might be important in explaining the "closeness" of the products, the second dimension had no evident interpretation. With the provision of so little attribute-related information, the conclusion is that multidimensional scaling is not a viable approach to uncover the competitive attribute structure of a market.

The benefit of using cluster analysis to uncover competitive partitions is that the procedures are well recognized. There are several difficulties in using cluster analysis. First, the type of cluster analysis must be selected, along with the appropriate measure of between-cluster distances. Second, the degree of agglomeration in defining partitions is judgmental. Third, the hierarchy of attributes is deduced by examining the attribute composition of clusters at each level of agglomeration. Thus, different investigators could find various attribute structures based on their selection of method, judgment of when a new cluster should be combined to an existing cluster and how to assign attributes to clusters. However, for the experimental data analyzed in this study, hierarchical cluster analysis was able to identify a meaningful attribute structure.

The proximity matrix used in the entropy method is the product variant switching matrix. The main premise of the hierarchical structure is that the lower the observed switching across attribute states, the stronger the bond among variants within a partition. As each level is determined, the degree of switching within higher level partitions is evaluated for the remaining attributes. A significant advantage of the process is that it is entirely automated. The method takes full advantage of the known attribute composition of each variant in the product set. There are no subjective decisions to make so the analysis will always be consistent.

The study has shown that two methods are capable of using the same data to identify the same competitive structure of a simple market. Before judging them equal in this quest, the two methodologies need to be applied to real data, where there are often many attributes of interest and hundreds of product variants. To be able to identify the hierarchical structure in such cases requires large-scale diary panel data that are often readily available. Such an investigation is the subject of future research.

REFERENCES

Carter, J and Silverman, F, "An Empirical Approach to Market Partitioning: Application to the Cigarette Market," Journal of Targeting, Measurement and Analysis for Marketing, 12 (2004): 366-378

Cooper, D. and Emory, C., Business Research Methods. Irwin, 1995

Gardner, M.., "Advertising Effects on Attributes Recalled and Criteria Used for Brand Evaluations," Journal of Consumer Research, 10 (1983): 310-318

Garner, W. and McGill, W., "The Relationship Between Information and Variance Analysis," Pschometrika, 21 (1956): 219-228

Green, P., Analyzing Multivariate Data, Dryden Press, 1978

Kalwani, M. and Morrison, D., "A Parsimonious Description of the Hendry System," Management Science, 23 (1977): 467-477

Lehmann, D., "Judged Similarity and Brand-Switching Data as Similarity Measures," Journal of Marketing Research, 9, (1972): 331-334

Kumar, A. and Sahi, C., "Confirmatory Analysis of Aggregate Hierarchical Market Structures> Inferences from Brand Switching Behavior," Journal of Marketing Research, 26 (1989): 444-453

Rao, V. and Sabavala, D., "Inference of Hierarchical Choice Processes from Panel Data," Journal of Consumer Research, 8 (1981): 85-96

Shannon, C. E., "A Mathematical Theory of Communication," Bell System Technical Journal 27 (1948) 379-423

John C. Carter, Pace University, Lubin School of Business, New York, New York, USA

Fred N. Silverman, Pace University, Lubin School of Business, New York, New York, USA

TABLE 1: PRODUCT DESCRIPTIONS

Variant     Brand     Color     Size

A             a         x        r
B             c         x        t
C             a         y        u
D             b         y        s
E             c         x        r

TABLE 2: SWITCHING MATRIX

                                       TO

                A           B           C           D           E
            (a, x, r)   (b, y, s)   (c, x, t)   (c, x, r)   (a, y, u)

A
(a, x, r)      10           1           0           1           5

B
(b, y, s)       2          11           1           2           1

C
(c, x, t)       1           0           9           5           2

D
(c, x, r)       1           1           6          13           0

E
(a, y, u)       5           1           2           1          19

TABLE 3: DISSIMILARITY MATRIX

        A         B         C         D         E

A     1.526     4.102     4.467     4.363     3.607
B     4.102     0.000     4.458     4.184     4.385
C     4.467     4.458     1.681     3.160     4.206
D     4.363     4.184     3.160     1.808     4.541
E     3.607     4.385     4.206     4.541     2.109

FIGURE 1: STIMULUS COORDINATES FROM SPSS

                               Dimension
Stimulus     Stimulus
Number       Name            1           2

1            A            -1.3730     -0.0055
2            B            -0.2092     -1.5489
3            C             1.1927      0.6716
4            D             1.3064     -0.2148
5            E            -0.9168      1.0976

FIGURE 4: AGGLOMERATION SCHEDULE

Agglomeration Schedule

                      Cluster Combined

Stage     Cluster 1     Cluster 2     Coefficients

1             3             4            3.160
2             1             5            3.607
3             1             2            4.244
4             1             3            4.370

                     Stage Cluster Firs
                          Appears

Stage     Cluster 1     Cluster 2      Next Stage

1             0             0              4
2             0             0              3
3             2             0              4
4             3             1              0

FIGURE 6: CLUSTER-ATTRIBUTE RELATIONSHIP

      Brand     Color     Size

C       c         x        t
D       c         x        r
A       a         x        r
E       a         y        u
B       b         y        s