Application of structured decision-making tools to defense acquisition.

The Defense Acquisition System is heavily dependent upon quality decision making. The application of structured decision-making tools to Defense acquisition problems can significantly assist the decision maker in the analysis of complex decisions, particularly those involving uncertainty, risk, and multiple objectives. Decision analysis and operations research are structured decision-making tools that can aid the decision maker in avoiding biases, documenting decision methodologies, and making group decisions. Overall, the systematic application of structured decision-making tools can significantly increase a decision maker's insight into the complex decisions that are characteristic of the Defense Acquisition System.

**********

Defense acquisition decisions are often of extremely high importance and consequence, as the lives of U.S. Armed Forces members and the people they protect may depend on the quality of those decisions. Decision analysis and operations research are two different structured decision-making methodologies that can be employed to significantly improve the quality of decision making and problem solving, as well as provide the decision maker with greater insights into the decision at hand. Decision analysis accentuates the decision maker's objectives, preferences, and attitudes towards risk (Goodwin & Wright, 2004). Operations research emphasizes system understanding and the formulation of a mathematical model of the system (Winston, 1994). The nature of the decision may indicate whether decision analysis or operations research is most applicable, but in many cases the application of more than one technique may help the decision maker view the problem from multiple perspectives. The methodologies often complement one another, providing the decision maker with significant insight into the decision at hand (Modjeski, 2004).

UNSTRUCTURED VERSUS STRUCTURED DECISION MAKING

Decision makers develop a personalized set of decision-making tools and strategies over time based on their experience and education. When faced with a decision, decision makers employ a strategy that they believe to be the most applicable based on the situation. Characteristics of the decision, such as urgency, importance, consequence, and available information, all affect a decision maker's choice of strategy. For common decisions of low importance and consequence, decision makers typically employ unstructured decision-making tools and methods, called heuristics (Gigerenzer, Todd, & ABC, 1999). Heuristics may provide satisfactory courses of action but often do not provide the optimal course of action in a given decision (Goodwin & Wright, 2004). For some simple Defense acquisition decisions, such as the purchasing of copier paper, the minimal complexity and low consequence of the decision may not warrant the time and effort required to employ a structured decision-making tool. For the purchasing of copier paper, a decision maker might utilize a heuristic strategy where he/she will rank the various attributes of available vendors in order of importance and choose the vendor that provides the highest value on the most important attribute (Goodwin & Wright, 2004). Should the lowest purchase price be the most important attribute, corresponding to the objective of minimum cost to the government, the decision maker will choose the vendor that provides copier paper that meets minimum requirements at the lowest price. If two vendors provide copier paper at the same lowest price then the decision maker will choose the vendor that provides the most value on his/her next most important attribute, such as delivery time.

Most Defense acquisition decisions are significantly more complex than the purchasing of copier paper, and therefore the use of unstructured heuristics is not appropriate. In Defense acquisition, decision makers are typically faced with complex decisions involving multiple objectives. As indicated in the Federal Acquisition Regulations, Part 1.102 (2005):

   The vision for the Federal Acquisition System is to deliver on a
   timely basis the best value product or service to the customer,
   while maintaining the public's trust and fulfilling public policy
   objectives. Participants in the acquisition process should work
   together as a team and should be empowered to make decisions
   within their area of responsibility.

For Defense acquisition decisions of high importance and consequence, a decision maker should employ a compensatory, structured decision strategy to arrive at an optimal course of action versus an unstructured heuristic strategy. Unlike heuristic strategies, which are noncompensatory, a compensatory strategy requires the decision maker to not only rank the importance of multiple objectives and their associate attributes, but to make trade-offs between various attributes. Poor performance by a decision option on one attribute might be offset by superior performance on several another attributes (Goodwin & Wright, 2004). In the case of the copier paper example, the decision maker might choose to purchase copier paper from a more expensive vendor based on the vendor's history of superior delivery times, responsiveness, and product quality. Decision analysis and operations research are compensatory, structured decision-making tools that can provide the decision maker with significant insight into complex defense acquisition decisions.

DECISION ANALYSIS

Robert T. Clemen (1996), Associate Professor of Decision Sciences, Duke University, provided the following summary of the objectives of decision analysis and outlined the decision analysis process as shown in Figure 1:

   I subscribe to the notion that the objective of decision analysis
   is to help a decision maker think hard about the specific problem
   at hand, including the overall structure of the problem as well as
   his or her preferences and beliefs. Decision analysis provides both
   an overall paradigm and a set of tools with which a decision maker
   can construct and analyze a model of a decision situation ... the
   purpose of studying decision-analysis techniques is to be able to
   represent realworld problems using models that can be analyzed to
   gain insight and understanding. It is through that insight and
   understanding--the hoped-for result of the modeling process--that
   decisions can be improved.

[FIGURE 1 OMITTED]

Decision analysis commences with a thorough identification of the problem and then places heavy emphasis on the subjective judgment of the decision maker. The objectives of the decision maker along with his/her preferences are explored and evaluated during the process of decomposing and modeling of the problem. Decision analysis tools, including the Simple Multi-Attribute Rating Technique (SMART) and multi-attribute utility theory, are utilized to elicit value and utility functions from the decision maker as well as his/her attitudes towards risk (Goodwin & Wright, 2004). After the preferred alternative is identified, sensitivity analysis is conducted. During sensitivity analysis, the decision maker investigates the dependencies of preferred solutions on the inputs obtained during the elicitation and modeling stages of the decision analysis process prior to implementation of the chosen alternative (Goodwin & Wright, 2004). Employment of the decision analysis process can provide Defense acquisition decision makers with new insights into complex procurement decisions.

SMART DECISION ANALYSIS TOOL

The SMART provides the decision maker with a compensatory, structured analytical process for evaluating complex decisions that involve multiple objectives where uncertainty is not a factor (Edwards, 1971). The SMART's relative simplicity, speed of application, and transparency--i.e., easy for individual and group decision makers to understand--make the tool an extremely valuable asset to the decision maker. When compared to noncompensatory, heuristic-based decision methods, SMART can provide the decision maker with a significantly greater understanding of complex Defense acquisition decisions (ODPM, 2004; Goodwin & Wright, 2004).

The first stage of SMART is to identify the decision maker. In the case of a new Defense weapons system procurement, the acquisition team members are the decision makers. In the second stage, the alternative courses of action are identified. For a simplified weapon system procurement example, the alternatives may be limited to the procurement of weapon system 1 or weapon system 2. In stage 3, the attributes that are relevant to the decision are identified. For this example, the attributes are determined to be cost, development schedule, destructive power, accuracy, and speed of employment. A value tree is displayed in Figure 2 (Goodwin & Wright, 2004).

[FIGURE 2 OMITTED]

In stage 4, values for the performance of weapon system 1 and weapon system 2 on each individual attribute are computed. As all of the attributes for the weapon system procurement example can be denoted with quantifiable variables, Table 1 provides the variables and their associate value functions for weapons systems 1 and 2. In each case the preferred variable is assigned a value of 100 and the least preferred variable is assigned a value of 0 (for problems with additional alternatives, values between 100 and 0 would also be assigned as appropriate) (Goodwin & Wright, 2004).

In stage 5, the decision maker is asked to determine weights for each attribute to reflect his/her preferences between the attributes. The SMART (Edwards, 1971) model is a linear additive model where the total value for each decision option (weapons systems 1 and 2) is the sum of the values assigned to each individual attribute for the option multiplied by its respective weight, as shown in Equation 1 (ODPM, 2004):

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] Equation 1

The weights are determined to reflect the decision maker's preferences between attributes. A simple procedure would be to have the decision maker rank the attributes in order of preference. Unfortunately, a simple ranking method might provide too much weight to an attribute that is important to the decision maker but has little bearing on the decision at hand (Goodwin & Wright, 2004). For example, if the difference in length of development schedule between the two alternatives was only one week, the importance of development schedule on this specific decision may be negligible, but the importance to the decision maker of development schedule as an attribute may be significant. To avoid such an issue, the decision maker is encouraged to assign swing weight to each attribute. Edwards & Barron (1994) referred to the use of the SMART method with swing weights as SMARTS, which stands for SMART with Swings (ODPM, 2004).

To determine the swing weights, the decision maker is asked to rank the attributes based on the swing from least to most preferred variable of each attribute versus the swing from least to most preferred variable of the other attributes. The attribute with the lowest importance is assigned a weight of 0 and the one with the highest is assigned a weight of 100. The remainders are assigned intermediate values and then all the values are normalized (Goodwin & Wright, 2004). The results for the weapon system procurement example are shown in Table 2.

Equation 1 can now be utilized in stage 6 to determine the overall values for weapons systems 1 and 2 portrayed in Table 3. Table 3 demonstrates how attribute swing weights and attribute values can be combined using Equation 1 to provide insight to the decision maker regarding the weapon system procurement decision. Purchase of weapon system 2 received a higher total value than that received by weapon system 1. After making a provisional decision in step 7 to purchase weapon system 2 based on the results in Table 3, the decision maker should complete step 8 of SMARTS. In step 8, sensitivity analysis is completed to determine how the results of the analysis might change based on changes in the values and weights provided by the decision maker. Step 8 is very important (and often neglected) as the results obtained will provide the decision maker with an enhanced understanding of the problem and better confidence in the final Defense acquisition decision (Goodwin & Wright, 2004).

UTILITY THEORY DECISION ANALYSIS TOOL

Although more complicated than SMARTS, utility theory provides the decision maker with a compensatory, structured analytical process for evaluating complex decisions that involve one or more objectives where uncertainty and risk are factors in the decision. A utility function can be derived from the decision maker's attitude towards risk and utilized to provide significant insight into the decision at hand (Goodwin & Wright, 2004). According to the University of Michigan Decision Consortium (2004):

   Utility theory is an attempt to infer subjective value, or
   utility, from choices. Utility theory can be used in both decision
   making under risk (where the probabilities are explicitly given)
   and in decision making under uncertainty (where the probabilities
   are not explicitly given).

To continue with the weapon system procurement example, single attribute utility theory can be utilized to evaluate development schedule risk for weapons systems 1 and 2. In their work Theory of Games and Economic Behavior, John von Neumann and Oskar Morgenstern initiated both game theory and the theory of choice under uncertainty (Economics, 2004). Von Neumann and Morgenstern's (1944) theory of utility can be employed in the weapons system procurement example to elicit a utility function for the decision maker's attitude towards risk (Goodwin & Wright, 2004). The decision choices, either the procurement of weapon system 1 or 2, along with the probabilities of the development schedule outcomes based on the chosen weapon system, are represented in the decision tree shown in Figure 3.

[FIGURE 3 OMITTED]

Derivation of the decision maker's utility function for the possible development schedules shown in Figure 3 can be accomplished by presenting the decision maker with a series of hypothetical lotteries. First, the best outcome (12 months) is assigned a utility of 1.0 (u[12 months] = 1.0). Next, the utility of the worst outcome is assigned a utility of 0.0 (u[17 months] = 0.0). Intermediate utilities are calculated by conducting an elicitation session with the decision maker. For each intermediate outcome, the decision maker is asked to choose between various hypothetical lotteries which offer a specific percentage chance of achieving the best outcome and the corresponding percentage of achieving the worst outcome. Once the decision maker indicates indifference between a presented lottery and the actual outcome, the outcome is assigned the utility of the lottery (Goodwin & Wright, 2004).

For example, the decision maker is asked to choose between (a) the certainty of a 16 month development schedule or (b) engaging in a lottery where there is 50 percent chance of a 12-month development schedule and 50 percent chance of a 17-month development schedule. If the decision maker indicates that he/she is indifferent between the two choices then the utility of a 16-month development schedule is assigned the utility of that lottery. The remaining intermediate utilities can be determined in a similar fashion as shown in Table 4. The decision maker's utility function can then be graphed as shown in Figure 4.

[FIGURE 4 OMITTED]

The utility function in Figure 4 for the weapons system procurement example has a concave shape which is characteristic of a decision maker that is risk averse (Goodwin & Wright, 2004). The utility function can now be applied to the decision tree in Figure 3 to determine the expected utility for each decision option as shown in Equations 2 and 3 and summarized in Figure 5.

(0.2*0.7) + (0.8* 1.0) = 0.94 Equation 2

(0.1 *0.9) + (0.4*4.0) + (0.5*0.8) = 0.49 Equation 3

[FIGURE 5 OMITTED]

Based on the expected utilities shown in Figure 5, weapon system 1 appears to be the preferred option due to its higher expected utility, but prior to making a decision, the decision maker should perform sensitivity analysis and consistency checks on the provided data. By varying the information provided by the decision maker in the elicitation session, the sensitivity of the calculated expected utilities for each option to changes in the supplied data can be determined and evaluated. Consistency checks can determine if the utility function and calculated expected utilities accurately reflect the decision maker's attitudes toward development schedule risk (Goodwin & Wright, 2004).

As shown in the weapons system procurement example, single attribute utility theory can be a valuable tool for the decision maker when faced with complex decisions involving uncertainty and risk. Multi-attribute utility theory can be utilized to extend single-attribute utility theory to problems involving multiple attributes. Keeney and Raiffa (1976) proposed the following approach to derive multi-attribute utility functions to allow a decision maker to evaluate problems involving risk, uncertainty, and multiple attributes. If mutual utility independence exists between the multiple attributes, the following three-stage process can be utilized to obtain the multi-attribute utility function (Goodwin & Wright, 2004):

1. Obtain the single-attribute utility functions for each independent attribute.

u([x.sub.1], [x.sub.2]) =([x.sub.1]*u([x.sub.1])) + ([k.sub.2]*u([x.sub.2])) + ([k.sub.3]*u(([x.sub.2]) Equation 4

2. By using Equation 4, two single-attribute utility functions can be combined into a multi-attribute utility function (more than two single-attribute utility functions can also be combined into a multi-attribute utility function, but the equations are increasingly complex). In Equation 4, u([x.sub.1], [x.sub.2]) is the multi-attribute utility level when attribute 1 has utility level [x.sub.1] and attribute 2 has utility level [x.sub.2]. The [k.sub.1] and [k.sub.2] values are employed to weight the single-attribute values and are evaluated in a similar fashion to the swing weights under SMARTS, except that lotteries are utilized. The decision maker is asked to choose between the following options:

(a) A certain outcome where attribute 1 is at its best level and attribute 2 is at its worst level, or

(b) A lottery where there is a [k.sub.1] probability that both attributes will be at their best levels and a (1-[k.sub.1]) probability that both attributes will be at their worst levels.

The decision maker is then asked to choose between the following options:

(a) A certain outcome where attribute 2 is at its best level and attribute 1 is at its worst level, or

(b) A lottery where there is a [k.sub.2] probability that both attributes will be at their best levels and a (1-[k.sub.2]) probability that both attributes will be at their worst levels.

Equation 5 is then utilized to calculate k3.

[k.sub.3] = 1-[k.sub.1]-[k.sub.2] Equation 5

3. Complete consistency checks and sensitivity analysis on the multi-attribute utility function obtained in stage 2.

As was the case with SMARTS, the application of single- and multi-attribute utility theory can provide the decision maker with significant insights into complex decisions. The SMARTS, due primarily to its simplicity, can be an extremely valuable tool for employment in problems which do not involve uncertainty or risk. When uncertainty and risk are involved in a decision, as is often the case for Defense acquisition decisions, an understanding of single- and multi-attribute utility theory can also be a valuable asset to the acquisition decision maker.

OPERATIONS RESEARCH

The U.S. Department of Labor (2004) defines operations research as:

   Operations research and management science are terms that are used
   interchangeably to describe the discipline of applying advanced
   analytical techniques to help make better decisions and to solve
   problems. The procedures of operations research have given
   effective assistance during wartime missions, such as deploying
   radar, searching for submarines, and getting supplies where they
   were most needed.

Wayne L. Winston (1994) provided a similar definition of operations research as "a scientific approach to decision making, which seeks to determine how best to design and operate a system, usually under conditions requiring the allocation of scarce resources," and provided the seven step operations research analysis process shown in Figure 6.

[FIGURE 6 OMITTED]

As compared to the decision analysis process shown in Figure 1, which places a heavy emphasis on the subjective judgment of the decision maker, the operations research analysis process shown in Figure 6 places more emphasis on understanding the system, verifying the models, and formulating detailed mathematical models which incorporate risk profiles via probability distributions. A decision maker's subjective judgments, particularly those regarding risk, are not considered in operations research. As indicated by Dr. Richard Modjeski (2004):

   Decision makers often are critical of Operation Research methods
   for ignoring subjective judgments. Personal judgments are a
   critical part of making good decisions in decision theory.
   Decision makers often site Operations Research for being precisely
   wrong instead of approximately right. This refers to the tendency
   to solve the wrong problem with the right method.

In some cases, decision makers may even reject mathematical models developed under operations research that have been optimized for the objectives of the overall organization if the decision makers' personal preferences, objectives, and attitudes towards risk do not completely coincide with those of the organization. An example may be an acquisition manager who chooses a procurement alternative that is low in risk versus an alternative with greater risk and potentially higher benefits to avoid being associated with a possible project failure.

By viewing a complex problem from both decision analysis and operations research perspectives, a manager can gain significant insight into a decision as the two methodologies for handling risk complement one another (Modjeski, 2004). By employing both decision analysis and operations research, a risk-averse acquisition manager may be able to better balance his/her tendencies towards rejecting a new innovative alternative with significant risk and the DoD's goal of exploring new opportunities and emerging technologies. Decision analysis may identify an acquisition manager's risk aversion and assist in developing risk-reduction alternatives (Goodwin & Wright, 2004); whereas operations research may identify how a high-risk project fits into the DoD's overall military acquisition strategy that mitigates risk across numerous research and development projects throughout the Defense Acquisition System. A manager who understands how to employ both decision analysis and operations research methodologies in complex decision making will be much better prepared to strike a successful balance between minimizing risk and maximizing opportunities.

CONCLUSION

Quality decision making is critical to the success of the Defense Acquisition System. The lives of U.S. Armed Forces members and those they protect often depend on the quality of Defense acquisition decisions. When faced with complex Defense acquisition decisions of high importance, decision makers should employ compensatory, structured decision-making strategies to arrive at optimal courses of action versus heuristic strategies which provide only satisfactory solutions. Structured decision-making strategies, such as decision analysis and operations research can provide the decision maker with significant insight into Defense acquisition decisions. Application of multiple structured decision-making strategies can provide even greater insight by allowing the decision maker to view a decision from multiple perspectives as the strategies compliment one another. A decision-maker who takes the time to become proficient at applying multiple structured decision-making tools and strategies will be much better prepared to make quality Defense acquisition decisions, particularly when faced with complex decisions of high importance involving uncertainty, risk, and multiple objectives.

REFERENCES

Clemen, R. T. (1996). Making Hard Decisions (2nd ed.) Belmont, CA: Duxbury Press. Retrieved January 11, 2006, from http://faculty.fuqua.duke.edu/~clemen/bio/mhd/mhdpref.htm

Department for Communities and Local Government (DCLG) (2000, December). Multi-criteria analysis manual. Retrieved September 06, 2006, from http://www.communities.gov.uk/index.asp?id=1142251

Economics New School (2004, May 30). Oskar Morgenstern, 1902-1976. Retrieved January 11, 2006, from http://cepa.newschool.edu/het/profiles/morgenst.htm

Edwards, W. (1971). Social utilities. Engineering Economist, Summer Symposium Series 6, 119-29.

Edwards, W., & Barron, F.H. (1994). SMARTS and SMARTER: Improved simple methods for multi-attribute utility measurement. Organizational Behavior and Human Decision Processes (60), 306-325.

Federal Acquisition Regulation. (2005, March). Part 1.102. General Services Administration, Department of Defense, and National Aeronautics and Space Administration. Retrieved September 06, 2006, from http://www.acqnet.gov/far/current/html/Subpart%201_1.html#wp1130776

Gigerenzer, G., Todd, R M., & the ABC Research Group (1999). Precis of simple heuristics that make us smart. New York: Oxford University Press. Retrieved January 11, 2006, from http://www.bbsonline.org/documents/a/00/00/04/69/ bbs00000469-00/bbs.todd.html

Goodwin, P., & Wright, G. (2004). Decision analysis for management judgment (3rd ed.). New York: John Wiley & Sons.

Hardin, J. (2004, June 3). Virginia Commonwealth University. Math 327-Mathematical Modeling. Introduction to Operations Research, p. 5. Retrieved September 06, 2006, from http://www.courses.vcu.edu/MATH327/(under "Schedule" under "Lecture Slide").

Keeney, R. L., & Raiffa, H. (1976). Decisions with multiple objectives: Preferences and value tradeoffs. New York: John Wiley & Sons.

Modjeski, R. B. (2004, June 7). Florida Institute of Technology. MGT 5071: Decision theory class 9: Revising judgment in the light of new information & risk and uncertainty management. Unpublished lecture notes.

University of Michigan Decision Consortium (2004, May 22). Utility theory. Retrieved September 06, 2006, from http://www.lsa.umich.edu/psych/ decisionconsortium/Tutorials/utility_theory.htm

U.S. Department of Labor (2004, May 17). Bureau of labor statistics occupational outlook handbook. Washington, DC: Author. Retrieved January 11, 2006, from http://stats.bls.gov/oco/ocos044.htm

Van Dorp, J. R., & Mazzuchi, T. A. (2006). George Washington University. Slides for lectures Chapter 1, p. 8. Retrieved September 06, 2006, from http://faculty.fuqua.duke.edu/~clemen/bio/mhd/slides/htm

Von Neumann, J., & Morgenstern, O. (1944). The theory of games and economic behavior. Princeton, NJ: Princeton University Press.

Winston, W. L. (1994). Operations research: Applications and algorithms (3rd ed.). Belmont, CA: Duxbury Press.

LCDR John R. Gensure, USN (Ret.), is a Department of the Navy civil service employee at the Office of Technology Development and a retired Navy Engineering Duty Officer. He received a BS in physics from the U.S. Naval Academy, an MS in Electrical Engineering from the Naval Postgraduate School, and an MBA from the Florida Institute of Technology. An Acquisition Professional Community member with Program Management and SPRDE Level III certifications, he is an Acquisition Reform Certificate of Excellence recipient.

(E-mail address: [e-mail])

TABLE 1.
ATTRIBUTE VALUES FOR WEAPON SYSTEM PROCUREMENT EXAMPLE

                    Weapon System #1          Weapon System #2

                   Variable      Value       Variable      Value

Purchase Price    $1,000,000       0         $750,000       100
Development
Schedule          12 months       100       14 months        0
Destructive
Power             2,000 lbs       100       1,750 lbs        0
Accuracy
(Circular Error
Probable (CEP)      100 m         100         150 m          0
Speed of
Employment         1 minute        0          30 sec        100

TABLE 2.
WEIGHTS FOR WEAPON SYSTEM PROCUREMENT EXAMPLE

                                   Original    Normalized
                                     Swing       Swing
Atribute                Swings      Weights     Weights

Purchase Price         $250,000       100         40
Development Schedule   2 months       30          12
Destructive Power       250 lbs        0           0
Accuracy               50 meters      70          28
Speed of Employment     30 sec        50          20
Total                                 250         100

TABLE 3. PRODUCT OF VALUES AND WEIGHTS FOR WEAPONS
SYSTEM PROCUREMENT EXAMPLE

                     Weapon System #1             Weapon System #2

Attribute     Value    Weights   Product   Value    Weights   Product

Purchase        0        40         0       100       40       4000
Price
Development    100       12       1200       0        12         0
Schedule
Destructive    100        0         0        0         0         0
Power
Accuracy       100       28       2800       0        28         0
Speed of        0        20         0       100       20       2000
Employment
Total/100                          40                 60

TABLE 4.

UTILITIES FOR WEAPON SYSTEM PROCUREMENT EXAMPLE

u(16 months) = 0.5*u(12 months) + 0.5*u(17 months) = (0.5)*(1.0)
+ (0.5)*(0.0) = 0.5
u(15 months) = 0.7*u(12 months) + 0.3*u(17 months) = (0.7)*(1.0)
+ (0.3)*(0.0) = 0.7
u(14 months) = 0.8*u(12 months) + 0.2*u(17 months) = (0.8)*(1.0)
+ (0.2)*(0.0) = 0.8
u(13 months) = 0.9*u(12 months) + 0.1*u(17 months) = (0.9)*(1.0)
+ (0.1)*(0.0) = 0.9