Evaluating project proposals: a simple formula allows planners to score proposals in terms of the degree to which the available management resources meet the needs of the project.
Firms employ a wide range of performance measures when evaluating R&D performance (3). Likewise, different techniques are available to assist managers in assessing R&D project proposals and choosing among competing alternatives. These usually involve rating projects against a checklist of items such as technical merit, feasibility and potential for commercialization; statistical techniques are often employed to sort out correlation among the various criteria (4). Because many of these techniques are complex, having been developed primarily for use in large firms, it can be a daunting task for managers of small firms to apply them on a sustained basis; many seek assistance from outside supporting institutions (5). Consequently, there is a need for simpler qualitative evaluation techniques better suited to the needs and resources of SMEs (6).
In this article we introduce a straightforward method of evaluation based on a survey analysis of SMEs in Kanagawa Prefecture, Japan. It is based on the premise that the allocation of management resources has a major impact on the success rate of new R&D projects and is thus an important concern of R&D project managers. Managers of SMEs, keeping an eye on the changing competitive environment, strive to achieve an ideal distribution of resources that maximizes the success rate of new projects. The actual allocation of management resources, however, will necessarily be constrained by the limited stock of resources the firm currently has on hand. The difference between the actual and ideal distribution of resources provides an indication of the range of R&D projects that the firm could effectively undertake. Assuming that the success of a particular project would be inversely proportional to this difference, we assert that a quantitative indicator of this difference would provide a simple and useful tool to assist evaluators in prioritizing projects and weeding out those that stand little likelihood of success.
To develop a meaningful difference calculation requires identifying the appropriate management resources, as well as their relative weights that would constitute an "ideal" allocation; this benchmark allocation, we surmised, would vary from one firm to the next but would likely be similar for firms in similar industries.
Through preliminary discussions with SME managers, we were able to identify a set of eight categories, or "assessment items," by which managers of SMEs typically assess merits and demerits of R&D project proposals. Next, we asked managers of SMEs to assign weights to the eight assessment items that correspond to the relative emphasis given them during the project planning process; averaging the results across groups of similar firms in the survey yielded a list of "weighting coefficients" for each group. We express the results of the survey in the form of a simple "project proposal assessment equation" that can be used to calculate a comprehensive score for each project proposal. Comparing projects on the basis of their scores provides a simple way to identify and prioritize promising projects.
Suppose that the allocation of management resources corresponds to the relative importance of the project assessment criteria being used in the evaluation process. Also, assume that the merits of a given project plan can be expressed in the form of a "comprehensive preliminary project assessment score" [Y.sub.iR], whose value can range from zero to 100 and which is expressed in the form of equation (1):
(1) [Y.sub.iR] = ([R.sub.1] x [x.sub.i1]) + ...... + ([R.sub.n] x [x.sub.in])
Here, [R.sub.n], is the "weighting coefficient" associated with the [n.sup.th] assessment item (n = 8), whose value will vary depending on the business conditions of the industry (0 [less than or equal to] [R.sub.n] [less than or equal to] 100). The variable [x.sub.in] represents the preliminary assessment score (0 [less than or equal to] [x.sub.in] [less than or equal to] 1) that the evaluator assigns to the [n.sup.th] assessment item for the project proposal of the [i.sup.th] firm (i [less than or equal to] total number of plans being assessed); for each assessment item, this score is obtained by taking the amount of investment that evaluators believe the firm could allocate to the project and dividing it by the ideal amount of investment the evaluators believe would guarantee the project's success. Finally, the product [R.sub.n] x [x.sub.in] represents the weighted score given by firm i to assessment item n (0 [less than or equal to] [R.sub.n] x [x.sub.in] [less than or equal to] [R.sub.n]).
Weighting coefficients [R.sub.1] through [R.sub.n] express the benchmark weights of the respective items used in the preliminary assessment of R&D projects, this value being the maximum possible contribution from resource n to the total score Y. In principle, the value of each coefficient can be obtained through 1) weighted regression and correlation analysis; 2) experience; 3) arbitrary judgment of the supporting institution/evaluator in advance of the evaluation. In this study, we focus on 2), that is, on identifying through our survey instrument the weights that experienced project planners in SMEs regard on average as being ideal.
To illustrate, assume that a machinery firm wishes to evaluate and prioritize two R&D project proposals. Also assume that the evaluators have at their disposal the results of a survey of machinery firms that yielded the following average weights for three assessment items: management organization (45), R&D (30), competitive environment (25). Using these weights as a benchmark, the evaluators estimate for each proposal the degree to which they could realistically allocate resources to these three items, along with an approximation of the "ideal" investment that would be needed to guarantee the projects' success.
The results are shown in Table 1. For each project, the computed ratios [x.sub.1] and [x.sub.2], along with the weighting coefficients, are used with equation (1) to calculate comprehensive preliminary project assessment scores [Y.sub.iR], which in this case yield scores of 36.8 for project 1 and 44.6 for project 2. In this example, therefore, project 2 would be the preferred project for the firm to undertake.
As a starting point, we drew up a list of eight assessment items commonly used in preliminary project evaluation: management organization, R&D, production/sales, capital, competitive environment, project development, customer expectations, and contribution to society. These items are displayed in the left-hand column of Table 2. Next, we presented the list to managers from a sample of regional SMEs, asking them to assign to each item a weight ranging from 0 to 100 with the stipulation that the sum of the weights for all items must total 100.
The population targeted in the survey were firms based in Kanagawa Prefecture, Japan, that in the fall of 1997 had fewer than 300 employees and a capitalization of less than [yen] 100 million. From these, we drew a random sample of 17 firms known to be actively developing new products and commercializing new technology. For each sampled firm, we randomly selected one official from among those responsible for project planning. Sampled firms ranged from makers of measurement machinery, manufacturers of chemical detergent compounds, printed circuit design firms, and manufacturers of equipment that processes and transmits weather information. Numbers of employees ranged from fewer than ten to more than 200, and annual sales ranged from several million yen upwards to several billion yen.
Assuming that the weights associated with the various assessment items would likely vary across industries, we employed a means of distinguishing among different types of industries. In considering the economic characteristics of the industrial sector of each surveyed firm, we took into account both the scale of production facilities and the scope of operations. As shown at the top of Table 2, we divided the sample into four groups representing four "industry type clusters." These comprised manufacturers that mass-produce a wide range of products, manufacturers that produce a narrow range of specialized products in small quantity, service providers that offer a wide range of services, and service providers that offer a narrow range of specialized services. Firms in the survey were allocated to a cluster based on the type of good produced and configuration of its value chain.
What We Learned
Results of the survey are summarized in Table 2 and the chart below. The averaged weighting coefficients fall into three broad categories. Those in the ballpark of 18-15 include management organization (including management system, qualities of CEO, leadership), customer expectations, and R&D (including R&D organization, novelty of new products and commercialization capabilities). Project development (timing of R&D results, payback time, commercial prospects), capital (including financial position, expected future profits), and production/sales fall into the 12-11 range. Average weighting coefficients in the 9-7 range include competitive environment (position with respect to rivals, regulatory factors bearing on commercialization, location factors) and contribution to society (including effects on employment, influence on other industries). Looking at the results in the aggregate, the assessment item most heavily emphasized is management organization (average weighting value of 18.43). For firms in type cluster B (few specialized products), however, this item is in second position, and the range of values (17.06-20.88) is somewhat dispersed. This suggests that the relative importance managers attach to the eight assessment items varies by type of good produced.
Overall, the second most heavily emphasized assessment item is customer expectations (16.63), followed by R&D (14.96). Among manufacturers in type clusters A and B, however, customer expectations ranks third. As with management organization, the range of values is quite broad (14.71-18.12), which again suggests a link between assessment strategy and industry type. R&D receives considerably less emphasis among providers of varied services. Contribution to society received, on average, the smallest weighting coefficient (7.55), which suggests that the societal impact of new projects is not a major concern of most firms.
Returning to equation (1), which describes the preliminary project assessment score [Y.sub.iR] as a weighted average of all assessment items, we now substitute the values for the weighting coefficients obtained from the survey. Suppose that the project evaluator at company i assigns to each of eight assessment items the scores [x.sub.i1] to [x.sub.i8], respectively. Using rounded figures for the average weighting coefficients obtained for all firms (see chart), we can rewrite equation (1) in the form of equation (2) as follows:
(2) [Y.sub.iR] = (18 x [x.sub.i1]) + (15 x [x.sub.i2]) + (11 x [x.sub.i3]) + (11 x [x.sub.i4]) + (9 x [x.sub.i5]) + (12 x [x.sub.i6]) + (17 x [x.sub.i7]) + (7 x [x.sub.i8])
The benchmark coefficients shown here are averages; R&D managers would likely wish to use values that have been derived for their own industry or that they themselves have determined on the basis of experience. Using Equation 2, the total score for each project can be easily obtained after the evaluator has calculated the fraction of available resources demanded by the project ([x.sub.i]) for each assessment item.
More Important Than R&D
It is noteworthy that R&D is not the most heavily weighted factor in the assessment process for three of the four industry type clusters. More important than R&D are management organization and customer expectations. Consequently, we conclude that a project strong in R&D potential will have difficulty earning a high score in the absence of strong positive impact on management resources and a demonstrable link to customer needs.
On average, the most important factor stressed in the evaluation process is management organization. In other words, insufficient organizational resources will drive down the total score for a project by a greater margin than would a comparable lack of resources in other areas. Firms will thus tend to favor projects for which sufficient organizational resources can be readily tapped.
Next in priority are customer expectations and R&D. Evaluators will thus favor projects for which sufficient R&D capabilities can be easily accessed and which deliver demonstrable value to the firm's customers. Firms in type cluster A (manufacturers of many varied goods) give relatively greater emphasis to societal contribution than firms in other groups; customer expectations, by contrast, receive less emphasis (though the standard deviation is quite large). For firms in type cluster B (manufacturers of few specialized products), R&D is the most important assessment item; product development receives comparatively less emphasis. This is because R&D tends to be an important source of competitive advantage for producers of specialty or niche market goods. On the other hand, R&D is far less important for firms in type cluster C (providers of many varied services). For retailers and other mass service providers, many of which face rapid product cycles, project development, the competitive environment and availability of capital have comparatively greater bearing on the choice of projects.
Type D firms (providers of few specialized services) show a broader range of responses: some emphasize management organization, while others do not. For those that stress management organization, we surmise that the attitude, enthusiasm and vision of managers are vital to a project's success; for others, organization may be more fluid and thus less relevant to the smooth execution of a project.
Production/sales and competitive environment also figure less prominently in the evaluation process; being specialty service providers, these firms generally have lower production costs, and competitors tend to be fewer in number than is typical for other industries.
These results are consistent with those obtained from similar studies of preliminary project assessment methodology that have been applied in project administration, standards setting and other areas. A close parallel can be found in the European Quality Adjudication Model (7). In the EQA model, which was applied in the British construction industry, nine items comprise the list of evaluation criteria. These items, also displayed in the right-hand column of Table 2, correspond closely to the assessment items used in our study. For example, combining leadership and personnel administration in the EQA model yields a factor close in meaning to the management organization item used in this survey.
Comparing the weighting coefficients from this study with those from the EQA model, we see that R&D receives greater weight in our study, while the EQA model places greater emphasis on project development and customer expectations. This stands to reason in light of the fact that our study focuses on R&D project planning, while project development and customer needs would clearly be of paramount concern in the construction industry. Although the weighting of the various evaluation criteria would be expected to differ from one industry to another, the actual criteria used in the evaluation process are strikingly similar.
In Japan, 70 percent of firms decide on R&D themes solely through a process of internal deliberation. This most often is based on the ringi system, a process of decision making used by most Japanese firms in which consent of all concerned individuals is obtained prior to implementation by circulating a proposal throughout the organization. Only 20 percent of firms employ a scoring method, typically evaluating each assessment item on a five-point scale (8).
Given the low average success rate of R&D (about 3 percent) and commercialization (about 18 percent) in Japanese firms (9), project evaluators are under pressure to develop more objective and precise techniques for assessing R&D projects. Many SMEs seek support from public institutions to assist them in project planning and evaluation. Most commonly consulted are the public research centers (kosetsushi) that are funded by regional governments for the purpose of providing technical and training support to local SMEs. However, although these centers possess a wealth of information and expertise on R&D, they typically lack staff with experience managing or commercializing R&D. Firms thus face growing pressures to introduce and develop this expertise in-house. The method proposed in this study is simple enough to enable small companies to perform their own assessments, fine-tuning the weighting coefficients to suit the particular characteristics of their industry.
To sum up, this study has introduced a simple technique for evaluating new R&D project proposals, which is based on the premise that project planners prefer to move forward on projects for which the most critical management resources are in abundant supply. By identifying appropriate assessment criteria and determining their relative importance, we introduced a formula that planners can use to score project proposals in terms of the degree to which available management resources meet the needs of the project. In our survey of regional SMEs, when management resources for new projects are categorized into eight discrete items, managers responsible for new project planning weight each item in the following descending order of importance: management organization, customer expectations, R&D, project development, capital, production/sales, competitive environment, and contribution to society. The results suggest that the values of the weighting coefficients, as well as their order, depend on the industry being evaluated and the purpose of the evaluation.
The analysis rests on the hypothesis that success or failure of a project will be determined by the difference between the "ideal" allocation of management resources needed to complete the project successfully and the actual allocation that would be possible given current resource constraints. Substituting the values for the weighting coefficients obtained through our survey of SME managers yields a formula that computes a comprehensive preliminary assessment score for a given project. The higher the score, the greater the likelihood that the project will succeed. Because firms in different industries emphasize different criteria in the assessment process, the weighting coefficients may differ across industries; we use average values to simplify the explanation. In addition, managers can always alter the weights or add new assessment criteria as circumstances dictate. The technique should be especially useful in comparing and prioritizing a large number of project plans within a short period of time.
Table 1.--Evaluation Process for Comparing Two R&D Project Proposals Project Proposal 1 Weighting Coefficient Ratio Assessment Item [R.sub.n] Ideal Possible [x.sub.in] Management organization 45 60 30 0.5 R&D 30 40 10 0.2 Competitive environment 25 30 10 0.33 Project Proposal 2 Ratio Assessment Item Ideal Possible [x.sub.in] Management organization 80 30 0.38 R&D 20 10 0.5 Competitive environment 20 10 0.5 Table 2--Summary of Survey Results Showing Averages of Weights Managers Assigned to Each Project Proposal Assessment Item (Listed in descending order of importance of average weights for all industry type clusters) Industry All Type Cluster Industries A B Type of Good Many varied Few Produced products specialized products Industry Example Finished Manufactured manufactured parts, goods (autos, specialized drugs, home fabricators elec), construction Assessment Item Average Average Average weight weight weight (Standard (Standard (Standard deviation) deviation) deviation) Management 18.43 17.06 17.82 organization (8.64) (6.43) (9.87) Customer 16.63 14.71 15.76 expectations (8.30) (7.17) (7.78) R&D 14.96 15.88 18.06 (5.24) (4.61) (6.75) Project 11.60 10.59 9.00 development (4.66) (4.50) (4.07) Capital 11.19 11.18 11.06 (4.87) (4.03) (5.40) Production/sales 10.85 13.82 12.52 (5.38) (5.01) (6.04) Competitive 8.80 7.18 8.71 environment (4.22) (2.64) (5.40) Contribution to 7.55 9.59 7.06 society (4.64) (7.22) (3.00) Industry Reference Type Cluster C D Example (7) Type of Good Many varied Few specialized Criteria of Produced services services EQA Model Industry Example Small retailers, Venture British eating/drinking business, construction establishments, software industry distribution development consulting, printing Assessment Item Average Average weight weight (Standard (Standard deviation) deviation) Management 17.94 20.88 Leadership, organization (5.44) (11.41) 10+ People management, 9=19 Customer 17.94 18.12 Customer expectations (8.75) (9.34) satisfac- tion, 20 R&D 9.12 16.76 Policy and (6.00) (8.39) strategy, 8 Project 12.82 14.00 Business development (5.63) (4.28) results, 15 Capital 12.65 9.88 Resources, 9 (5.72) (4.09) Production/sales 9.53 7.53 Processes, 14 (6.6) (3.26) Competitive 12.53 6.76 People environment (5.15) (2.94) satisfac- tion, 9 Contribution to 7.47 6.06 Impact on society (4.47) (2.23) society, 6
References and Notes
(1.) Teece, David and Pisano, Gary. "The Dynamic Capabilities of Firms: An Introduction." In Technology, Organization, and Competitiveness: Perspectives on Industrial and Corporate Change, ed. Giovanni Dosi, David J. Teece, and Josef Chytry. New York: Oxford University Press, 1998, pp. 193-212.
(2.) Pierce, James K. "The Art of Creating a Flexible R&D Organization." Chemtech February 1998, pp. 6-11.
(3.) Werner, Bjorn M. and Souder, William E. "Measuring R&D Performance--U.S. and German Practices." Research * Technology Management May-June 1997, pp. 28-32.
(4.) Kurokawa, Susumu. "Make-or-Buy Decisions in R&D: Small Technology-Based Firms in the United States and Japan." IEEE Transactions on Engineering Management 44, pp. 124-134 (1997).
(5.) Kostoff, R. N. "Federal Research Impact Assessment: State-of-the-Art." Journal of the American Society of Information Science 45, pp. 428-440 (1994).
(6.) Lema, N. M. and Price, A. D. "Benchmarking: Performance Improvement Toward Competitive Advantage." Journal of Management in Engineering, January/February 1995, pp. 28-37.
(7.) Davies, Paul. "Help or Hazard? Total Quality Management, April 1993, pp. 35-38.
(8.) Yamanouchi, Akio. Kenkyu hyokaron koen roku 50: Kigyo ni okeru kenkyu hyoka no shiten (Research Evaluation Lecture 50: Points on Research Evaluation in Firms). Tokyo: National Institute of Science and Technology Policy, 1997, p. 34. The right-hand column of Table 2 presents weighted average assessment scores from a study of managers' evaluation of research project proposals based on a survey of firms in the British construction industry conducted by the Institute of Science and Technology Policy. The criteria used in this study were somewhat different from ours, although we judged them to be comparable. The scores show that our results are consistent with other research that uses a similar methodology. Both studies, for example, suggest that management and customer expectations weigh most heavily in decisions regarding R&D project selection.
(9.) Wakoh, Hikoji, "Tokkyo no jisshi kakuritsu no yosoku ni tuite" (On Estimating the Probability of Patent Implementation). Anzen kogaku, pp. 203-206 (1997).
Hikoji Wakoh is senior researcher in the Planning Division of the Kanagawa Industrial Technology Research Institute (KITRI), a publicly funded research and testing laboratory located in Ebina City, Kanagawa Prefecture, Japan. He has six years of experience as a new-project explainer and consultant to small and medium-sized enterprises in Kanagawa Prefecture, especially in the areas of technology and product development. He also works as consultant engineer for the Science and Technology Agency. Author of more than 40journal articles in engineering, he holds a Ph.D. in chemistry from Tokyo University. email@example.com
Steven Collins is associate professor in the Global Studies and Science, Technology & Environment Program at the University of Washington, Bothell Campus. From September 1997 through August 1998, he was STA/NSF research fellow at the Kanagawa Industrial Technology Research Institute, where he studied technology development in small and medium-sized companies with co-author Hikoji Wakoh. From September through December 1998, he was visiting researcher at Japan's National Institute of Science and Technology Policy. He holds a Ph.D. in government and foreign affairs and a B.S. in chemical engineering, both from the University of Virginia. firstname.lastname@example.org
|Printer friendly Cite/link Email Feedback|
|Author:||Wakoh, Hikoji; Collins, Steven|
|Article Type:||Brief Article|
|Date:||Nov 1, 2001|
|Previous Article:||Commercializing discontinuous innovations: an approach based on venture capital may prove the best way of dealing with the creative destruction...|
|Next Article:||From a twinkle in the eye to a blockbuster drug: the story of Celebrex[R] holds lessons for R&D leaders everywhere. (The IRI Medalist's Address).|