Using expert panels to evaluate new technologies.

ABSTRACT

In this paper, we review alternative strategies for evaluating the commercial potential of new technologies. We examine the role of expert panels and illustrate the robustness of this approach through case studies describing the application of engineering, medical, and information technology expert panels in technology commercialization studies in the aerospace and biomedical industries.

1. INTRODUCTION

The challenge of evaluating new technologies is encountered by Technology Managers in private-sector and public-sector companies. Extensive information is available on new technologies from the United States Patent and Trademark Office (USPTO, 2003) and the equivalent patent information centers in Europe (European Patent Office, 2003) and Japan (Japanese Patent Office, 2003). Although the literature contains several metrics for evaluating technologies (Geisler, 2002) and planning new business ventures (Timmons and Spinelli, 2003), few guidelines exist for the systematic evaluation of new, unproven technologies.

Studies on technology commercialization in the USA have indicated relatively poor performance levels (Leonard-Barton et al, 1994) with many technology-based inventions never moving beyond the conception stage (Jolly, 1997). Several companies have realized that successful technology commercialization efforts require ongoing concurrent collaboration, co-operation and teamwork among research scientists, engineers, designers and many business functions during all phases of the commercialization process (Nevens et al, 1990). As shown in Figure 1, successful commercialization efforts typically evolve through constant iterations between a new technological capability and a market need (Jolly, 1997). Of vital importance is the critical, objective and unbiased assessment of the proposed technology as it progresses through the phases of the technology development cycle.

[FIGURE 1 OMITTED]

Expert panels can provide a flexible and robust approach to evaluating new technologies and can be used to elicit unbiased evaluations from domain experts, develop prototypes or conduct experiments with new technologies. They have been used to evaluate, validate, and recommend actions on issues of importance in business, engineering, healthcare, law, science, education, government, and politics. Expert panels are particularly useful when historical data are limited, future events are likely to invalidate the results of historical analysis, or ethical and moral factors significantly impact the decision. These circumstances are often encountered when new technologies are being deployed.

In this paper, alternative strategies for evaluating the commercial potential of new technologies are examined. In addition, we define the role of expert panels and introduce a framework for deploying the expert panel process. Furthermore, we illustrate the robustness of this strategy through case studies describing the application of engineering, medical, and information technology expert panels in technology commercialization studies in the aerospace and biomedical industries.

2. STRATEGIES FOR EVALUATING NEW TECHNOLOGIES

2.1 Alternative Evaluation Strategies

Several strategies have been adopted to facilitate the evaluation of new technologies. These include Software Programs, Surveys/Studies, Prototyping/Experimentation, Task Forces/Special Committees, The Delphi Process, and Expert Panels. These are discussed in the following sections.

Software Programs

Software programs incorporating techniques such as simulation, forecasting, and technology mapping have been used to evaluate the commercial potential of new technologies. These programs facilitate the assessment of commercial development parameters (i.e. demand factors, competing technologies, resource requirements, regulatory factors and general constraints, and synergistic technologies), technology growth potential, and implications of commercialization over time. Software such as TechMonitoring[TM] (SRI Consulting Business Intelligence, 2003), Explorer[TM] (NTTC, 2003), and the TOP Index (ITRC, 2003), enable organizations to receive global, objective, and real-world analysis, identify new business opportunities, validate internal knowledge and communicate ideas, improve technology portfolio decisions, and provide a bridge between technical and business domains.

Surveys/Studies

Surveys and case studies have been used to develop valuable quantitative and qualitative information which may be used to demonstrate the feasibility of implementing a technology and establish a commercialization model. For example, the Interstate Technology and Regulatory Cooperative (ITRC) Working Group of the Global Network of Environment and Technology uses state case studies to detail innovative approaches for overcoming regulatory barriers (USDOT, 2003).

Prototyping/Experimentation

Many public and private institutions have used prototyping tools in conjunction with experimentation and other evaluation techniques to assess the application value and business potential of new technologies. Prototypes can be developed to examine the merits and risks of technologies at critical milestones in the technology development cycle.

Task Forces and Special Committees

Task forces and special committees (e.g. advisory committees, review committees, stakeholder review boards, decision and policy bodies, etc.) are special groups of representatives appointed to advise on critical issues (GAC, 1999). There is usually a time limit for reaching a conclusion and resolving a particular issue. People in these groups are usually considered "experts". They allow open discussion and the incorporation of a wide range of opinions from a broad cross-section of interests in the exploration of issues. They may employ a neutral facilitator or leader to encourage participation and feedback and often require significant resources.

2.2 Expert Panels

2.2.1 Expert Panel Processes

Expert panels are used in a variety of different fields and areas--including business, health care, law, science, education, government, politics, etc.--for the purpose of evaluating, validating, and recommending actions on issues of importance. The use of expert panels in these processes has been identified as a practical and serviceable approach to evaluation (Seskin et al, 2002). Expert panels have a long history of successful applications, in both their structured and unstructured formats and are an appropriate choice for evaluating new technologies.

Traditional expert panel approaches include the gathering of experts to reach a consensus, often in workshops or conference settings. The expert panel approach therefore requires a method of reconciling differences of opinion. Several suggestions on the refinement of the traditional expert panel approach can be found in the literature. One study, (Press, 1978) proposes using qualitative controlled feedback as opposed to quantitative controlled feedback so that respondents are not "pressured" toward consensus. In this approach, individual opinions are incorporated in building a consensus. This allows all opinions to be observed and noted with no pressure.

Although all opinions are accounted for in the final decision, one disadvantage of this method is that it disregards expert biases. The open forum style leaves room for "noise" and possible influence of more noted and reputable experts. Research shows that in group environments, participants' status affects group dynamics (Shelton, 1999). Therefore, the expert panel process must be carefully structured to ensure full participation from the panelists and limit the impact of domineering personalities.

2.2.2 The Delphi Method

The Delphi Method developed by senior scientists at the RAND Corporation in the 1950s, seeks to align the survey/questionnaire responses of experts, thereby forming a consensus on items of interest. Although the method was developed mainly as a forecasting technique (Levary and Han, 1995), its use today focuses on achieving the convergence and consensus of opinion.

The procedure elicits opinions from different persons (usually experts) without bringing them together in face-to-face confrontation. This process involves a "controlled feedback approach", usually requiring several iterations, or rounds of questioning. Summary feedback from anonymous responses to questionnaire items are gathered and distributed to all participants. The participants are asked to respond to the same questionnaire items again, with this information before them. The purpose here is for respondents to discover the degree of agreement or disagreement that appears in each item. Theoretically, the number of rounds is not pre-determined. Rather, the process continues until responses do not change distinctly. The final step is to determine the consensus arrived at after the iterations.

An important use of the Delphi method is in situations where each member of the panel is an expert on some aspect of a problem, but no panelist is an expert on the whole problem. Another traditional characteristic of the Delphi method is the use of autonomous feedback.

The "Delphi" method is a well-documented method of group decision-making which utilizes expert opinion. However, the process is very time-consuming and can go on for weeks and/or months. Also, questions have been raised about its reliability in forming valuable consensus of opinion. Critics have argued that since Delphi participants face an artificially induced pressure toward consensus, there is no guarantee that the consensus arrived at is correct (Press, 1978).

2.2.3 Virtual Expert Panels

Where traditional panels are unable to facilitate effective decision-making, the virtual expert panel can serve as an alternative method of analysis. As do all "expert panels", the virtual expert panel uses a group of experts for the purpose of evaluating, validating, and recommending actions on issues of importance. However, the virtual expert panel gets rid of the "noise" associated with open discussion and introduces a new model of autonomous feedback, asynchronous timing, and no concern for spatial proximity (distributed participants). Table 1 compares the strengths and weaknesses of the virtual panel, asynchronous and distributed expert panels, to those of other currently used forms of expert panels, which are characteristically synchronous and co-located.

Like a typical expert panel, the virtual panel will not meet several times, nor will it have the need for several iterations and will incorporate all opinions in the decision-making process. Feedback may not be used, especially in the form of controlled feedback. However, it may be used in the manner mentioned above--given as a summary of expert views to check for accuracy of reporting. Table 2 exhibits these elements of the virtual expert panel, making both comparisons with and distinctions from other expert panel processes discussed in this paper. Note that the virtual panel is most like the Delphi method--a popular method used in forming consensus--but considers providing accuracy by incorporating all expert opinions. This also ensures that valuable information is not lost through sifting out of expert views.

Additionally, virtual expert panels can be used as an enhancement to other more traditional technology evaluation methods. This combination can further facilitate the commercialization process and can possibly help to develop a robust framework that will reduce the time and money spent on reviewing new technologies.

3. A FRAMEWORK FOR EXPERT PANEL EVALUATION

Expert panels can serve as an effective approach to evaluate new technologies. However, careful attention must be paid to procedural issues and problems as these can critically affect the success of the expert panel (Seskin et al, 2002). The following four phases show the proper implementation to develop the framework for deploying expert panels:

* Define Problem Scope

* Create Expert Panel

* Develop Expert Panel Process

* Document Results

Figure 2 summarizes the framework adopted in our study to deploy expert panels to evaluate new technologies.

[FIGURE 2 OMITTED]

Phase 1: Define Problem Scope

* Define Problem

* Determine Clear Objectives

* List Tasks (milestones)

Phase 2: Create Expert Panel

* Locate and Contact Potential Experts

** Professionals, Scientists, and Doctors

* Determine Experts from Potential Pool

** Provide a pool of diversified talent

Phase 3: Develop Expert Panel Process

* Determine Expert Interaction

** Groupware, Conferences, Video Conferences

* Establish how Information will be Provided

** Electronic files, Hard Copies

* Select Strategy/Model of Analysis

** Delphi Method, etc.

* Determine Focus of Evaluation

* Convergence and Form Consensus of Opinion

Phase 4: Document Results

* Rank List of Potential Applications/Competing Technologies

* Inform Applicants of Analysis/Evaluation

4. CASE STUDIES

4.1 Case Study #1--Engineering Panel

Over the years, NASA-Langley Research Center (NASA LaRC) has developed over 170 patents and has been interested in commercializing the more promising ones. One of the technologies identified for review was an Ice Thickness Gauge Technology (NASA Tech Briefs, 1995) which addresses the hazards posed by ice build-up on aircraft, buildings and other structures. The Ice Thickness Gauge Technology developed by NASA-Langley Research Scientist, Dr. Leonard Weinstein and registered as US Patent No. 4,766.369 dated August 23, 1988 was developed to measure the build-up of ice over a detector. This technology offered promise because of its ability to distinguish between ice and water and to measure the thickness of ice with a high level of accuracy. Its versatility suggested good potential for deployment in a wide range of applications. These factors prompted its selection as a candidate for a preliminary technology commercialization study.

A virtual expert panel of twelve experienced engineering professionals was formed and a survey instrument developed and administered to the panelists to identify potential applications for the ice gauge technology and to provide an assessment of its technological capabilities. The expert panel provided an overall rating for the suggested applications and assessed them in terms of market potential, ease of manufacture, performance, and serviceability using a Staple scale ranging from - 4 (poor) to + 4 (excellent).

Table 3 provides profiles of the three case studies discussed in this section.

The twelve application areas suggested by the expert panel were:

* Automobile Skid Avoidance

* Ice Monitoring in Orchards

* De-Icing of Roadways

* Low Power De-Icing Systems

* Automobile Window De-Icing

* Water Pipes

* Lake Applications

* Underside of Aircraft

* Vehicle/Boat Windshields

* Bridges

* Tractor Trailers

* Electric Pylons & High-Tension Wires

The average ratings received were:

Market potential       + 2.7
Serviceability         + 1.3
Ease of Manufacture    + 2.1
Performance            + 1.9
Overall                + 2.4

The results re-inforced the findings in complementary market research studies which indicated that the ice gauge device had the potential to satisfy a number of customer-driven needs across several diverse applications and industries. However, the aircraft industry seemed to offer the best potential for establishing a commercial venture. This was the focus adopted in developing a financial model to assess the financial viability of commercializing this technology.

4.2 Case Study #2--Medical Panel

A small biomedical engineering start-up company was engaged in the development and marketing of a home and self-care health system which incorporated an ultra intelligent, vital signs--real time monitor to operate as a wireless baby monitor. This system would enable the user to interpret readings of vital signs and heart rate when the monitor was in use and, as a preventive measure, track and notify guardians in the event of an emergency. One application of the technology was to reduce the incidence of Sudden Infant Death Syndrome (SIDS), commonly called crib death.

To assess the market potential of the home and self-care health system developed, an expert panel of medical professionals was convened. The panel consisted of medical professionals with extensive practical experience in medicine and/or in medical education, with a range of 19 to 35 years of professional medical experience in a variety of specializations, including Pediatrics, Obstetrics/Gynecology, Rheumatology, and Internal Medicine/Public Health. Professionals in these fields--which are either directly of indirectly involved with the problem of SIDS and other related diseases/disorders or hospital care--can provide valuable and objective data for the purposes of the survey.

The survey instrument developed and administered to the panel included three sections. Section One addressed the market potential of the home monitoring system. Questions centered on identifying potential applications of the home monitoring system, including SIDS prevention and elderly/adult health care applications. A 5-point semantic scale (1-Low; 3-Moderate; 5-High) was used to assess the clinical value that the device could provide for each potential application. In addition, questions about the Safety and Effectiveness of current home monitoring devices were included. The panelists were also asked to give an overall rating for the proposed bio-monitoring system.

Section Two allowed the expert panelists to offer any general comments pertaining to both the proposed home monitoring system as well as similar devices. Specifically, participants were asked to give their professional opinion on any deficiencies that the proposed system and similar home monitoring devices had in providing efficient monitoring of at-risk SIDS patients. Section Three recorded the biographical information of the panelists.

Analysis of the survey results summarized in Figure 3 revealed the following:

* SIDS is one of the top-ranked applications for the home monitoring system securing a mean rating of 4.25/5 and ranking third among a list of 13 potential applications

* Other applications with high clinical potential include Apnea, Bradycardia, and an Apparent Life Threatening Event (ALTE). These applications had ratings of 4.5/5, 4.5/5, and 4.25/5, respectively. Each of these disorders is related to the sudden death of infants.

[FIGURE 3 OMITTED]

These results suggest that the proposed system may have widespread application as a monitoring device for those infants who are "at-risk" for sudden death (including SIDS), which may follow breathing pattern irregularities and heart rate fluctuations during sleep. The panel assigned encouraging ratings to the home monitoring system in the areas of Safety (3.75/5) and Effectiveness (3.0/5) and assigned a promising Overall Rating (3.75/5). These findings provided important feedback to the BioMed management team in their efforts to launch a successful business venture and led to complementary consumer market research studies to identify the profile of likely purchasers of the system.

4.3 Case Study #3--Information Technology Panel

Biomed Inc., the biomedical start-up company profiled in Case Study #2, was also interested in obtaining unbiased, expert evaluation of a new technology identified for licensing (Patent Application #60/352,096, Ridley 2002). The system was unique not only in its ability to continuously monitor an infant but also in its potential to be made affordable. The system was capable of continuous data monitoring and multivariate data analysis. It is designed to collectively support diagnosis, decision making, and corrective action. The technology's ability to cross correlate two or more related variables will provide a competitive edge over previous patents which monitor only one variable at a time, e.g. it can monitor the saturation of oxygen in the blood and the heart-rate simultaneously. The opportunity to provide the market with an affordable, highly dependable, multi-faceted communication and monitoring device has spawned several competing technologies. Our search of the patent database at the United States Patents and Trademark Office yielded the following list of 16 patents that could serve as primary competitors for the technology to be licensed by BioMed:

[FIGURE 3 OMITTED]

Of these, the four identified as the most competitive based on Invention Date and Technical Features were:

* Jackson Pat. # 6,047,201 (Apr. 2000): This uses a battery operated sensor (pulse oximeter) to collect data from an infant, and transmits the data via the 900MHz band to a battery operated monitor.

* Halleck et al Pat. # 5,549,113 (Aug. 1996): This system provides intermittent data transmission over a range of 90 feet.

* Zimmerman et al, Pat. #5,199,439 (Aug. 1999): This patent describes software that will require expensive and dedicated medical prescription sensors/monitors.

* Kim, Pat. # 5,505,199 (April 1996): This system requires a pulse oximeter, motion detector, and video camera all connected to a central unit.

To provide an objective comparison of our proposed technology with the top four competing patents, we commissioned a Technology Panel of engineers with expertise in Software Applications Engineering, Information Technology, and Integrated Product Development. We provided profiles of each technology extracted from the US Patents and Trademarks Office and directed the expert panelists to additional information sources for supplementary information. Our survey instrument asked the panelists to rate the technologies using a five-point semantic scale (1-Poor; 5-Excellent) using seven evaluation criteria. As shown in Table 5 below, the technology identified for licensing by BioMed Inc., (Patent Application #60/352,096, 2002) received the highest overall score (28.6), indicating that it did have a potential competitive advantage over its major competitors.

5. CONCLUSIONS

Expert panels can provide a robust, flexible approach to technology evaluation and can be used at various stages of the technology commercialization process. Successful deployment of expert panels however requires the development of a structured decision-making process, the provision of adequate information, and the development of appropriate decision-making tools. The impact of expert panels can be greatly enhanced through the use of Group Decision Support Systems (GDSS) which can facilitate information dissemination, enable timely feedback, stimulate intra-group collaboration, and permit the deployment of group decision-making techniques such as Scoring Models, Analytic Hierarchy Process (AHP), and Quality Function Deployment (QFD).

TABLE 1: ASYNCHRONOUS & DISTRIBUTED PANELS VS. SYNCHRONOUS &
CO-LOCATED PANELS

                        STRENGTHS              WEAKNESSES

ASYNCHRONOUS            Freedom from time      * Less interactive
&                       and place              * Can be slow and
DISTRIBUTED             * Allows flexible        cumbersome
                          access               * Requires high
(e.g. Virtual Expert    * Gives time for         motivation to
Panel, Delphi Panels)     reflection and         participate
                          reference
                        * Can enable
                          interaction among
                          large numbers
                        * Technologies
                          generally more
                          robust and less
                          bandwidth hungry.

SYNCHRONOUS             * Interactive          * Inflexible use
&                       * Timely                 of time
CO-LOCATED              * Sense of audience    * Dependent on
                        * Allows rapid           fast robust,
(e.g. NIH Consensus,      feedback,              connectivity
Nominal Group             turn-taking, and
Technique, Task           negotiation
Forces/Special          * Allows synergy
Committees)             * Brainstorming

Adapted from University of Plymouth @ Seale-Hayne "Typology of
computer-mediated Communications" (Based on Goodyear and Steeples,
Creating shareable representations of practice, ALT-J 6 (3) (1998))

TABLE 2: COMPARISON OF FOUR EXPERT PANEL PROCESSES

                                          TASK
VIRTUAL EXPERT      TRADITIONAL      FORCES/SPECIAL    DELPHI EXPERT
PANELS             EXPERT PANELS       COMMITTEES          PANELS

Questionnaire/    Face-to-Face      Face-to-Face      Questionnaire/
Survey Format     interaction/Open  Interaction/Open  Survey Format
                  Discussion        Discussion
Moderate                                              Very structured
structure         Limited           Limited           and systematic.
                  structure and     Structure and
Autonomy (no      unsystematic      unsystematic      Autonomy (no
face- to-face                                         face-to-face
interaction)      Instant           Many meetings     interaction)
                  feedback.         are required      Reduction of
Reduction of                                          noise in
"noise" in        Very Few          Specific time     consensus.
consensus.        Meetings          frame for
                                    decision          Asynchronous &
Asynchronous &    Incorporates all                    Distributed
Distributed       Opinions          Requires
                                    extensive         Time Consuming
Not Time          Possible          resources
Consuming         influence of      (i.e. time,       Process
                  opinions through  money, sponsor
Incorporates All  "noise"           support,          Relatively
Opinions                            expertise.)       Inexpensive
                  Provides
Relatively        accuracy of       Facilitator is    Controlled
                  expert views.     needed to         Feedback
Inexpensive                         encourage
                  Diversity         feedback and      Possible Forced
Provides                            discussion.       Consensus
accuracy of
expert views.                       iversity (may be  No Guarantee
                                    limited)          on "Right"
May or may not                                        Consensus
use feedback.
                                                      Diversity
No instant
feedback
Diversity

TABLE 3: SUMMARY OF CASE STUDIES USING EXPERT PANELS TO EVALUATE NEW
TECHNOLOGIES

              CRITERIA                 Case #1

Technology    1. Company        NASA
                 Sponsor        National
                                Aeronautical
                                Space Agency

              2. Technology     Ice Detection Gauge
                 Description    Monitor Ice Build-up
                                on Surfaces

              3. Industry       Aerospace
               - Primary        Monitor Ice Build-up
                   Application  on Aircraft Wings

              4. Focus of       Assess Market
                   Evaluation   Potential of
                                Technology

Expert Panel  5. Expert Panel   -- 12 Engineering
                 Composition    Professionals
                                -- 15 + years'
                                experience

              6. Expert Panel   -- Asynchronous
                 Process        -- Distributed

              7. Information    -- Technology Profiles
              Provided          -- Glossary of Terms
                                -- Secondary Infor-
                                   mation Sources

              8. Results        Ranked list of
                                Potential
                                Applications

              CRITERIA                 Case #2

Technology    1. Company        BioMed Inc
                 Sponsor        Biomedical
                                tart-up Company

              2. Technology     Bio-Monitoring
                 Description    System
                                Monitor Vital Signs

              3. Industry       Biomedical
               - Primary        Monitor Vital Signs in
                   Application  Infants

              4. Focus of       Assess Utility of
                   Evaluation   Technology

Expert Panel  5. Expert Panel   -- 4 Medical
                 Composition    Scientists/Doctors
                                -- 25 + years
                                experience

              6. Expert Panel   -- Asynchronous
                 Process        -- Co-located

              7. Information    -- Technology Profiles
              Provided          -- Glossary of Terms
                                -- Secondary Infor-
                                   mation Sources

              8. Results        Ranked list of
                                Potential
                                Applications

              CRITERIA                 Case #3

Technology    1. Company        BioMed Inc
                 Sponsor        Biomedical
                                start-up Company

              2. Technology     Bio-Monitoring
                 Description    System
                                Monitor Vital Signs

              3. Industry       Biomedical
               - Primary        Monitor Vital Signs
                   Application  In Infants

              4. Focus of       Assess
                   Evaluation   Competitiveness of
                                Technology

Expert Panel  5. Expert Panel   -- 4 IT Professionals
                 Composition    -- 10 + years
                                experience

              6. Expert Panel   -- Asynchronous
                 Process        -- Distributed

              7. Information    -- Technology Profiles
              Provided          -- Glossary of Terms
                                -- Secondary Infor-
                                   mation Sources

              8. Results        Ranked list of
                                Potential Competing
                                Technologies

TABLE 4: LONGLIST OF COMPETING PATENTS (SOURCE: US PATENTS &
TRADEMARKS OFFICE

 Patent No.      Invention        Inventor
                   Date

1. 6,041,201    Apr. 2000     Jackson
2. 6,011,477    Jan. 2000     Teodorescu et al
3. 5,941,820    Aug. 1999     Zimmerman et al
4. 5,564,433    Oct. 1996     Thornton
5. 5,549,113    Aug, 1996     Halleck et al
6. 5,505,199    April. 1996   Kim
7. 5,491,474    Feb. 1996     Sunii et al
8. 5,490,523    Feb. 1996     Issacson et al
9. 5,215,098    Jun. 1993     Steinhaus et al
10. 5,199,439   Apr. 1993     Zimmerman et al
11. 4,844,086   Jul. 1989     Duffy
12. 4,625,733   Dec. 1986     Saynajakangas
13.             Apr. 1986     Hutchins et al
14. 4,545,388   Oct. 1985     Jahn et al
15. 4,519,395   May 1985      Hrushesky et al
16. 3,332.954   May 1967      Bell et al

TABLE 5 : AVERAGE RATINGS OF COMPETING PATENTS ASSIGNED BY
TECHNOLOGY PANEL

                                     Competing Patents

Technology Evaluation Criteria       Pat. #6,047,201  Pat. #5,549,113
                                     Jackson et al.   Halleck et al.

Q1 Versatility and Utility                 3.3              2.7
Q2 Effectiveness                           2.3              3.7
Q3 Compatibility in Hospital               2.7              3.7
Q4 Compatibility with Human Subject        3.0              3.3
Q5 Ability and Effectiveness in
   Infants                                 3.0              4.0
Q6 Marketability                           2.3              2.0
Q7 Uniqueness                              3.3              3.0
Q8 Difficulty in Reproducing or
   Improving                               2.0              3.0

               Total                      22.0             25.3

               Rank                         5                2

                                     Competing Patents

Technology Evaluation Criteria       Pat. #5,941,820  Pat. #5,505,199
                                        Zimmerman           Kim

Q1 Versatility and Utility                 3.3              3.3
Q2 Effectiveness                           3.0              2.3
Q3 Compatibility in Hospital               3.0              2.7
Q4 Compatibility with Human Subject        3.3              3.7
Q5 Ability and Effectiveness in
   Infants                                 3.0              2.3
Q6 Marketability                           2.3              1.7
Q7 Uniqueness                              3.0              4.0
Q8 Difficulty in Reproducing or
   Improving                               2.7              2.3

               Total                      23.7             22.3

               Rank                         3                4

                                     Competing Patents

Technology Evaluation Criteria           Patent
                                       Application
                                       #60/352,096

Q1 Versatility and Utility                 4.0
Q2 Effectiveness                           3.7
Q3 Compatibility in Hospital               3.7
Q4 Compatibility with Human Subject        3.3
Q5 Ability and Effectiveness in
   Infants                                 4.0
Q6 Marketability                           2.3
Q7 Uniqueness                              3.7
Q8 Difficulty in Reproducing or
   Improving                               2.7

               Total                      27.3

               Rank                         1

Rating Scale: 1 - Poor 3 - Good 5 - Excellent

ACKNOWLEDGEMENTS

The authors would like to thank the Faculty and MBA students in the School of Business and Industry, Florida A & M University for their collaboration on this project.

REFERENCES

Geisler, Eliezer, "The Metrics of Technology Evaluation: Where we stand and where we should go from here", International Journal of Technology Management, Vol. 24, No. 4, 2002

Timmons, Jeffry A. and Spinelli, Stepheni, New Venture Creation: Entrepreneurship for the 21st Century, 6th edition, Irwin/McGraw-Hill, 2004.

National Technology Transfer Center, (2003), Products and Services TOP Index Software, June 30 http://www.nttc.edu

United States Patent and Trademark Office, (2003), Patents, May 20, http://www.uspto.gov

European Patent Office, (2003), Patent Information Centres, August 8, http://www.epo.co.at

Japanese Patent Office, (2003), Industrial Property Digital Library, August 8 http://www.jpo.go.jp

SRI Consulting Business Intelligence. 2003. Technology-Mapping Service: Explorer[TM]. Available at http://www.sric-bi.com/marketing/Xplr/Xplrbrochure.pdf

National Technology Transfer Center, (2003), Products and Services, TOP Index Software, June 30 http://www.nttc.edu

"Overcoming Regulatory Barriers to Commercialization." The Global Network of Environment and Technology: Interstate Technology and Regulatory Cooperative Working Group (ITRC). State Best Practice #8, 1994. Available at http://www.gnet.org/government/stategov/1929.cfm. Sept. 23, 2003.

U.S. Department of Transportation, Federal Highway Administration. Public Involvement Techniques For Transportation Decision-Making. Available at http://www.fhwa.dot.gov/reports/pittd/cover.htm. November 18, 2003.

"GAC Urged to Refrain from Increasing Task Forces and Special Committees." General Assembly Council of the General Assembly of the Presbyterian Church (U.S.A.) News. June 18, 1999.

Levary, Reuven R. and Han, Dongchui "Choosing a technological forecasting method." Industrial Management. January 1995.

Press, S. James, "Qualitative Controlled Feedback for Forming Group Judgments and Making Decisions." Journal of the American Statistical Association. Vol. 73, No. 363, pp. 526-35, Sept. 1978.

Seskin, Samuel, Gray-Still, Katherine, & Boroski, John. "The Use of Expert Panels in Analyzing Transportation and Land Use Alternatives." Parsons Brinckerhoff Quade & Douglas, Inc. Portland, Oregon. April 2002

Shelton, Robert. "The Long Game: Creating Revolutionary Change Through Radical Innovation." Arthur D. Little's Technology and Innovation Management Practice. Palo Alto, California. 1999.

Jolly, Vijay K., Commercializing New Technologies, Harvard Business School Press, Boston, MA, 1997.

Leonard-Barton, Dorothy, Edith Wilson and John Doyle, "Commercializing Technology: Imaginative Understanding of User Needs", Harvard Business School Teaching Note #9-694-102, September 1994, Harvard Business School Publishing, Boston MA 02163.

NASA Langley Research Center, "Technology Transfer Process", http://tag.larc.nasa.gov/tag/tschtransfer.html, February 1999.

NASA Tech Briefs, The Digest of New Technology, Measuring Thickness of Ice When Liquid is Present, Vol. 19, No. 4, April 1995.

Nevens, T. Micahel, Summe, Gregory L. and Bro Uttal, "Commercializing Technology: What the Best Companies Do", Harvard Business Review, May-June 1990, pp. 154-162

Author Profiles:

Dr. Colin O. Benjamin is the Anheuser Busch Professor of Engineering Management in the School of Business and Industry at Florida A & M University. He has had several years of international teaching, industrial and consulting experience. He received a Ph.D. in Industrial Engineering from the University of the West Indies, an MBA from the Cranfield Institute of Technology, UK, and an MSc in Engineering Production and Management from the University of Birmingham, UK.

Elisha Archibold is an Information Systems Consultant and received her MBA from the School of Business and Industry at Florida A & M University. She has conducted several Technology Evaluation Studies and is currently involved in ERP systems implementation.

Dr. Tiki Suarez is an Assistant Professor of Information Systems in the School of Business and Industry at Florida A & M University. She has several years of teaching, research and consulting experience. She received her Ph.D. in Computer Science from Florida State University and BS/MS degrees in Computer Science from Clark Atlanta University.