Transformation of analytical tools: using portfolio analysis techniques in defense applications.
Quantitative measures are under development to assess the Department of the Navy (DON) portfolio of system acquisitions in order to improve business practices through better analytical tools and models. As a result, attention shifts from analyzing individual acquisition programs (now studied exhaustively) to analyzing a portfolio of systems as a whole, which is similar to the methodology employed as a best practice in the private sector. This macro view will give DON senior leaders valuable metrics metrics Managed care A popular term for standards by which the quality of a product, service, or outcome of a particular form of Pt management is evaluated. See TQM. for measuring risks and uncertainties of costs, capabilities, and requirements. Armed with these metrics, senior leaders can make better choices, among a set of plausible portfolios, to satisfy the Navy's national security objectives.
Early phases of the initiative identified and evaluated existing models and industry practices. Next, a subset of the current DON portfolio was selected by financial management and acquisition staff with which to test a methodology of portfolio analysis: Mine Countermeasures All methods for preventing or reducing damage or danger from mines. Also called MCM. , a diverse, representative system of programs. This pilot model is a multi-phase process that includes the following:
* Gathering life cycle cost data for the various systems that will be analyzed
* Establishing a scoring system Noun 1. scoring system - a system of classifying according to quality or merit or amount
classification system - a system for classifying things using subject matter experts to determine how effectively current and future systems match capabilities to requirements
* Developing a means to display results by which decision makers can examine risk-reward analysis and conduct trade-offs
The ultimate goal is to assess DON investments using portfolio analysis methodology.
According to according to
1. As stated or indicated by; on the authority of: according to historians.
2. In keeping with: according to instructions.
3. the Honorable Donald Rumsfeld, "What you measure, improves." In this regard, the Department of Defense (DoD) is quite adept at measuring the cost and the value of a specific program to fulfill a specified mission. Trade-offs are conducted and analyses of alternatives are studied. Sometimes, gap analyses are performed. But are such comparisons made program versus program? Are funding decisions made with an emphasis on leadership strategy and national objectives? Portfolio analysis is a promising method to improve DoD business practices by analyzing a portfolio of systems as a whole, rather than analyzing individual acquisition programs (Figure 1).
[FIGURE 1 OMITTED]
When the Honorable Richard Greco, Jr., Assistant Secretary of the Navy Assistant Secretary of the Navy (abbrev. "ASN") is the title given to certain senior officials in the U.S. Department of the Navy. They serve as chief assistants to the Secretary of the Navy (SECNAV). (Financial Management and Comptroller), entered his position in late 2004, he was asked by the Vice Chief of Naval Operations The Vice Chief of Naval Operations (VCNO) is the top deputy to the Chief of Naval Operations, in the United States Navy. Appointed by the President of the United States under authority of an act of March 5, 1948 (62 Stat. 67). to consider devising methods that could be used to analyze programs and better inform resource decision making early in the Planning, Programming, Budgeting and Execution process. This request was the beginning of an effort to analyze common and best practices in government and in the private sector and then to use and augment those techniques in a DON construct. The Naval Cost Analysis Division, directed by Ms. Wendy Kunc, was given the job. "We were excited at the prospect of developing these new analysis tools and were challenged at the magnitude of this daunting daunt
tr.v. daunt·ed, daunt·ing, daunts
To abate the courage of; discourage. See Synonyms at dismay.
[Middle English daunten, from Old French danter, from Latin task."
Portfolio analysis is the art and science of allocating scarce resources to satisfy strategic objectives. In literature, this form of analysis is described as a dynamic decision process, a resource allocation resource allocation Managed care The constellation of activities and decisions which form the basis for prioritizing health care needs process, or a manifestation man·i·fes·ta·tion
An indication of the existence, reality, or presence of something, especially an illness.
(man´ifestā´sh of a business strategy. In government, as well as in the private sector, portfolio analysis helps senior management determine where and how to invest for the future. In short, it is a technique to determine how to best spend limited dollars.
Portfolio management is characterized by the following:
* Uncertain and changing information
* Dynamic environment
* Multiple goals and strategic considerations
* Interdependence in·ter·de·pen·dent
Mutually dependent: "Today, the mission of one institution can be accomplished only by recognizing that it lives in an interdependent world with conflicts and overlapping interests" among projects
* Multiple decision makers and locations
As projects are analyzed, they usually are found to be in different stages of completion (for example, technology development, system development and demonstration, or production). While projects often are designed to fulfill multiple strategic goals and often are highly interdependent in·ter·de·pen·dent
Mutually dependent: "Today, the mission of one institution can be accomplished only by recognizing that it lives in an interdependent world with conflicts and overlapping interests" , they still compete against one another for scarce resources.
Both in government and in the private sector, portfolio analysis necessarily is prospective in nature, dealing with future events, opportunities, and costs. Information often is uncertain or, at worst, highly unreliable. The decision environment is also dynamic due to changing threats or requirements, as well as to changing status of projects. In addition, portfolios are constantly evolving as new information becomes available and as new projects are added or removed from the set.
Three basic goals of portfolio analysis are described in literature: value maximization Value Maximization
Increases in owners' wealth achieved by maximizing of the value of a firm's common stock. , balance, and strategic direction. For value maximization the focus is return on investment (ROI (Return On Investment) The monetary benefits derived from having spent money on developing or revising a system. In the IT world, there are more ways to compute ROI than Carter has liver pills (and for those of you who never heard of that expression, it means a lot). ) and the likelihood of success. For the private sector, ROI for an individual project might be expressed as net present value divided by investment dollars. For the military, the return may be a future stream of military capability divided by the investment.
In achieving a balance within a portfolio, a number of parameters often are considered:
* Short term versus long term
* High risk versus low risk versus sure bets
* Product categories versus technologies
* Development versus production versus basic research
* Production versus maintenance
Leaders of best-practicing organizations use these considerations to ensure that the projects are selected to meet long-term organizational goals, are tied directly to the organization's fundamental goals, and are "on strategy."
To maximize value, that value first must be measured, calculated, or deduced. It is instructive in·struc·tive
Conveying knowledge or information; enlightening.
in·structive·ly adv. to examine one of the more highly regarded financial models in the private sector for estimating a project's value (Figure 2, page 30).
[FIGURE 2 OMITTED]
The Expected Commercial Value (ECV ECV External Cephalic Version
ECV Enhanced Call Verification (ADT Alarm Services)
ECV Extracellular Volume
ECV Expanded Capacity Vehicle
ECV Electrical Cardioversion
ECV Expected Commercial Value
ECV Essential Climate Variable ) is based on decision-tree analysis. In this model, ECV considers future streams of earnings, probabilities of technical and commercial success, and costs. The variables in the equation are stochastic By guesswork; by chance; using or containing random values.
stochastic - probabilistic , not necessarily random but highly uncertain, and some more so than others. Costs are fairly well understood, but earnings are much less so. In practice, subject matter experts determine a "best guess" at strategic importance, probability of technical success, and probability of commercial success by completing scorecards.
Applying this commercial construct to a national security setting presents several significant challenges. Instead of producing future streams of earnings, DoD produces flows of military capability. Figure 3 (page 30) shows one method for modeling flows of military capability.
[FIGURE 3 OMITTED]
The National Security Strategy of March 2005 identifies four strategic objectives and eight required operational capabilities of U.S. military forces. The expected military value, in one of many constructs, is a function of the strategic importance of a project, the degree to which the capability is desired, as well as probabilities of technical and operational success. It is important to note that the variable for Strategic Importance is influenced directly by strategic objectives; similarly, the degree of capability desired is influenced by key operational capabilities.
While the private sector uses a common metric (that is, dollars) to determine value, there is no commonly defined metric for value across DoD programs. As a result, military value is extremely difficult to determine and must be subjective. With any subjective measure, the impact of special interests must be minimized and the results displayed in a readable, easy-to-understand format.
A best-practice means to display results of portfolio analysis employs a risk-reward bubble diagram, as shown in Figure 4 (page 31). The two axes axes
[L., Gr.] plural of axis. The straight lines which intersect at right angles and on which graphs are drawn. Usually the horizontal axis is the x-axis and the vertical one the y-axis. Called also axes of reference. are risk and reward that divide the chart into four quadrants. These quadrants, named in nautical nau·ti·cal
Of, relating to, or characteristic of ships, shipping, sailors, or navigation on a body of water.
[From Latin nauticus, from Greek nautikos, from terminology since the study was developed for the DON, represent the four categories in which each program is assessed: Watch Standing, Bravo Zulus Bravo Zulu is a naval signal, conveyed by flaghoist or voice radio, meaning "Well Done"; it has also passed into the spoken and written vocabulary. It can be combined with the "negative" signal, spoken or written NEGAT, to say "NEGAT Bravo Zulu", or "not well done". , Oysters, and Bilge Water bilge water
1. Water that collects and stagnates in the bilge of a ship.
2. Slang Nonsense.
Noun 1. .
[FIGURE 4 OMITTED]
The Watch Standing category (lower-left quadrant quadrant, in analytic geometry
1 In analytic geometry, one of the four regions of the plane determined by two lines, the x-axis and the y-axis. ) represents those programs with a high likelihood of success but with low or moderate reward. These projects are in abundance in any organization, and they usually contain many modifications, extensions, fixes and/or slightly different versions of the same endeavor.
Bravo Zulus are programs that are the potential stars, bow risk and high reward. Most firms desire more of these projects. Oysters are the long-shot projects. There is a high expected value Expected value
The weighted average of a probability distribution. Also known as the mean value. of the reward but with high risk. Usually they are programs that require some technical breakthrough in order to be successful, and many companies focus their attention on programs in this quadrant.
Lastly, every organization has at least one project of little value and high risk, those in the Bilge Water category. These usually have a strong advocate and are hard to kill. The size of each bubble is significant, representing resources. These resources could be an average annual cost or a total cost over some fixed amount of time, such as DoD's Future Years Defense Plan. In addition, some form of high-low-average measure may be represented to express cost uncertainty, as shown in the bubble in the lower right.
How are programs evaluated? The most common practice for evaluating value and risk is by using scorecards whereby a group of subject matter experts evaluate each program based upon common criteria (Common Criteria for Information Technology Security) An international standard process for defining security objectives and for evaluating compliance with those objectives. The Common Criteria have largely replaced the Trusted Computer Security Evaluation Criteria (TCSEC), the Canadian , and the results are statistically analyzed. In the Mine Countermeasures (MCM (MultiChip Module or MicroChip Module) A chip package that contains several bare chips mounted close together on a substrate (base) of some kind. ) pilot, a scoring conference was held in December 2005 with 15 subject matter experts gathered to score 40 different systems and 6 platforms using a model similar to that used at U.S. Special Operations Command A subordinate unified or other joint command established by a joint force commander to plan, coordinate, conduct, and support joint special operations within the joint force commander's assigned operational area. Also called SOC. See also special operations. , the Strategy-to-Task Model.
Common criteria must be established, based on the strategic goals of the organization, and projects must fulfill one or more objectives. In scoring capabilities and risks of MCM systems, a logical, rigorous, strategy-to-tasks approach is used, designed to link individual assets such as ships, sonars, and influence-sweep sleds to broad-based, macro national security objectives. By employing this approach, current and proposed systems are evaluated in terms of their contribution to the goals and priorities set, not by sponsors in a particular warfighting community, but by the most senior leadership in the Office of the Secretary of Defense The Office of the Secretary of Defense (OSD) is part of the United States Department of Defense and includes the entire staff of the Secretary of Defense. It is the principal staff element of the Secretary of Defense in the exercise of policy development, planning, resource and the DON. This builds into the process a guarantee that those systems that score highest, ceteris paribus Ceteris Paribus
Latin phrase that translates approximately to "holding other things constant" and is usually rendered in English as "all other things being equal". In economics and finance, the term is used as a shorthand for indicating the effect of one economic variable on [all other factors being equal], will be those that respond best to changes in strategy and priorities as defined by the Secretary of Defense, the Joint Chiefs of Staff, and the Chief of Naval Operations chief of naval operations
n. pl. chiefs of naval operations Abbr. CNO
The ranking officer of the U.S. Navy, responsible to the secretary of the Navy and to the President. . Figure 5 illustrates the architecture of this scoring system.
[FIGURE 5 OMITTED]
To employ this model, one first defines and weights a list of strategic tasks or requirements. In defining the strategic, macro-level tasks for Mine Warfare The strategic, operational, and tactical use of mines and mine countermeasures. Mine warfare is divided into two basic subdivisions: the laying of mines to degrade the enemy's capabilities to wage land, air, and maritime warfare; and the countering of enemy-laid mines to permit friendly , we referenced two documents: National Security Strategy (May 2005) and a recent Presidential security directive on the maritime domain. Then we asked subject matter experts to agree upon five MCM strategic tasks:
* Protect Operating Forces Those forces whose primary missions are to participate in combat and the integral supporting elements thereof. See also combat forces; combat service support element; combat support elements. Against the Threat of Sea Mines in the Littoral littoral /lit·to·ral/ (lit´ah-r'l) pertaining to the shore of a large body of water.
pertaining to the shore.
* Defend U.S. Ports and Coastal Approaches Against Sea Mines
* Maintain Mobility of Operational Forces in the Presence of Sea Mines
* Collect, Analyze, and Share Intelligence Related to the Worldwide Threat of Sea Mines
* Preserve Freedom of the Seas freedom of the seas: see seas, freedom of the. for Commercial Navigation in the Presence of Sea Mines
A problem immediately surfaces when ranking the importance of these tasks using subject matter experts. Namely, there is no flawless tool or technique to employ. As the great American mathematical economist Kenneth Arrow Kenneth Joseph "Ken" Arrow (born August 23, 1921) is an American economist, joint winner of the Nobel Prize in Economics with John Hicks in 1972, and the youngest person ever to receive this award, at 51. pointed out in his Ph.D. dissertation dis·ser·ta·tion
A lengthy, formal treatise, especially one written by a candidate for the doctoral degree at a university; a thesis.
1. in 1951, any technique that anyone can ever develop to rank-order preferences, other than using a dictator dictator, originally a Roman magistrate appointed to rule the state in times of emergency; in modern usage, an absolutist or autocratic ruler who assumes extraconstitutional powers. From 501 B.C. until the abolition of the office in 44 B.C., Rome had 88 dictators. , will violate at least one commonly accepted measure of fairness.
Nevertheless, the situation is not hopeless. Two imperfect imperfect: see tense. but highly regarded techniques that are employed are Condorcet's Method of Pairwise Comparisons and Borda's Count technique. The Special Operations Command uses the former in its strategy-to-task assessment model; major league baseball "MLB" and "Major Leagues" redirect here. For other uses, see MLB (disambiguation) and Major Leagues (disambiguation).
Major League Baseball (MLB) is the highest level of play in North American professional baseball. uses the latter annually in choosing the most valuable players.
Condorcet's technique matches one strategic task against each of the others, one at a time, in head-to-head competitions. A "one" is given for a win and a "zero" for a tie in each of the match-ups. The task with the greatest number of wins becomes the top preference. In the Borda Count The Borda count is a single winner election method in which voters rank candidates in order of preference. The Borda count determines the winner of an election by giving each candidate a certain number of points corresponding to the position in which he or she is ranked by each technique, numerical values or weightings are assigned to each of the tasks in a subject matter expert's vector of votes. We then sum the weightings across scorers. The task with the highest number of votes is the winner. As a side note, both the Condorcet and the Borda techniques use all of the information in a sample of scores, a very strong theoretical argument in favor of each. Other techniques do not possess this attractive property.
In a mock voting to test the methodology, the top-rated preference using both the Condorcet and the Borda techniques was Defend U.S. Ports and Coastal Approaches Against Sea Mines.
Indeed, the two different techniques yielded the same exact order of preferences for all five strategic requirements, giving some assurance that the sample voting was sound.
Next, a working group defined a list of operational and tactical tasks for Mine Warfare. We link tasks at one level to tasks at a higher level using a display and scoring technique shown in Figure 6, a sample matrix matching operational to strategic tasks.
[FIGURE 6 OMITTED]
For each cell in the matrix, we ask subject matter experts to determine the value of an operational task in meeting the requirements of a strategic task. Four responses are allowed:
* A critical operational task is a potential war stopper if not done.
* An essential operational task significantly mitigates risk, but is not a war stopper.
* A useful task enhances capability.
* A particular task may not be needed at all.
The values in the red cells in the Strategic Tasks column are weights obtained from the Borda Count technique and indicate the relative importance of the various strategic requirements. We multiply these values by entries in the cells of the matrix to generate a set of weights for each of the six operational tasks. This same approach is used in a second matrix (not shown here) for linking tactical tasks to operational tasks.
The penultimate pe·nul·ti·mate
1. Next to last.
2. Linguistics Of or relating to the penult of a word: penultimate stress.
The next to the last. step in the scorecard process is an evaluation of the capability of individual MCM systems. We evaluate each system in terms of its effectiveness and suitability on a low-to-high scale. Effectiveness, loosely stated, is the degree to which a system performs its mission, with speed included in the measure, an all-important metric in Mine Warfare. Suitability is the degree of availability, interoperability The capability of two or more hardware devices or two or more software routines to work harmoniously together. For example, in an Ethernet network, display adapters, hubs, switches and routers from different vendors must conform to the Ethernet standard and interoperate with each other. , maintainability, and so on. Effectiveness and suitability metrics are scored against each of the tactical tasks.
The desired outcome of scoring is to determine how well each system meets strategic, operational, and tactical tasks. Obviously, the more objectives the system satisfies, the higher the score. Higher-scoring projects will receive greater emphasis and quite possibly more resources for development and fielding.
Armed with the results of scoring and ranking, coupled with cost data, we are able to display the data in a format that is easily understood by senior leaders and decision makers. Ideally, the results could be manipulated near real-time so decision makers can conduct what-if questioning with answers provided in short order.
Results and Conclusions
Currently, NCAD NCAD National College of Art and Design (Irish college)
NCAD New Cumberland Army Depot
NCAD Noise Control and Acoustics Division (American Society of Mechanical Engineers) [Naval Cost Analysis Division] is concluding an analysis of a subset of Navy programs, Mine Countermeasures, as a proof of concept. "We are still at our early stages of development, but already we are showing ourselves to be proactive business partners to the programs and requirements communities, who have expressed not only enthusiasm but also critical input for the project," said Mr. Greco.
Several obstacles still remain. Mr. Robert Hirama, a key member of the Portfolio Analysis Team, stated, "One of the biggest hurdles has been the difficulty of scoring both dedicated systems and multipurpose mul·ti·pur·pose
Designed or used for several purposes: a multipurpose room; multipurpose software.
Adjective platforms given the complex interdependence Complex interdependence in international relations is the idea put forth by Robert Keohane and Joseph Nye that states and their fortunes are inextricably tied together. The concept of economic interdependence was popularized through the work of Richard Cooper. between them."
In addition, fair weighting of individual programs is problematic because when each program is evaluated in a scoring session, there are very few subject matter experts who have expertise in all programs, including leading-edge science and technology projects. Fewer still are those subject matter experts who can accurately assess the uncertainty of achieving technical success of a particular science and technology project that requires technical breakthrough. One final hurdle is the ability to report the analysis in an easily understood format that is acceptable to senior leaders, thereby providing useful information without becoming a full campaign analysis.
The results of portfolio analysis have the promise of giving senior leadership valuable metrics, including risks and uncertainties of costs, capabilities, and requirements, and of determining which portfolio to choose among a set of plausible portfolios for satisfying national security objectives. Mr. Greco believes that "the product NCAD is developing is groundbreaking. It will shape the way we look at investment decisions for many years to come."
(1) Arrow co-shared the Nobel Prize in Economics The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel, commonly called the Nobel Prize in Economics, is a prize awarded each year for outstanding intellectual contributions in the field of economics. in 1972 for this work. His conclusion is today calked "Arrow's Impossibility Theorem In voting systems, Arrow’s impossibility theorem, or Arrow’s paradox, demonstrates that no voting system can convert the ranked preferences of individuals into a community-wide ranking while also meeting a certain set of reasonable criteria when there are three ."
(2) As defined by the Joint Chiefs of Staff, Operational Effectiveness measures "the overall ability of a system to accomplish a mission when used by representative personnel in the environment planned or expected for operational employment of the system considering organization, doctrine, tactics, supportability, survivability sur·viv·a·ble
1. Capable of surviving: survivable organisms in a hostile environment.
2. That can be survived: a survivable, but very serious, illness. , vulnerability, and threat."
As defined by the Joint Chiefs of Staff, Operational Suitability is "the degree to which a system can be placed and sustained satisfactorily in field use with consideration being given to availability, compatibility, transportability, interoperability, reliability, wartime usage rates, maintainability, safety, human factors, habitability Fitness for occupancy. The requirement that rented premises, such as a house or apartment, be reasonably fit to occupy.
A Warranty of habitability is an implied promise by a landlord of residential premises that such premises are fit for human habitation. , manpower, logistics supportability, natural environmental effects and impacts, documentation, and training requirements."
Captain John Field is deputy director of the Naval Cost Analysis Division. He is a member of ASMC's Washington Chapter.
Brian Flynn Brian Flynn, (born 1955) is a Welsh football manager and former Welsh international player. He was a midfield player and, at 161cm (5'3"), one of the shortest professional football players of modern times. , PhD, is Head of the Economic Studies Branch of the Naval Cost Analysis Division.
Figure 6. Sample Matrix Linking Strategic MCM Tasks to Operational MCM Strategic weights from Operational Mine Borda Count Countermeasure Tasks Mine Perform Conduct Countermeasure Intelligence Q-Route Strategic Tasks Preparation Clearance of the Battlespace Defend U.S. 1.3 2 0 Ports and Coastal Approaches Against Sea Mines Collect, Analyze, 1.2 3 1 and Share Intel Related to the Worldwide Threat of Sea Mines Maintain Mobility 1.0 of Operational Forces Against the Threat of Sea Mines Protect Operating 0.9 Forces Against the Threat of Sea Mines in the Littoral Preserve Freedom 0.6 of the Seas for Commercial Shipping in the Face of Sea Mines Strategic weights from Operational Mine Borda Count Countermeasure Tasks Mine Conduct Conduct Countermeasure Port Operational Strategic Tasks Clearance Area Clearance Defend U.S. 1.3 0 Ports and Coastal Approaches Against Sea Mines Collect, Analyze, 1.2 1 and Share Intel Related to the Worldwide Threat of Sea Mines Maintain Mobility 1.0 of Operational Forces Against the Threat of Sea Mines Protect Operating 0.9 Forces Against the Threat of Sea Mines in the Littoral Preserve Freedom 0.6 of the Seas for Commercial Shipping in the Face of Sea Mines Strategic weights from Operational Mine Borda Count Countermeasure Tasks Mine Conduct Conduct Countermeasure Amphibious Follow-on Strategic Tasks Breaching Clearance Defend U.S. 1.3 Ports and Coastal Approaches Against Sea Mines Collect, Analyze, 1.2 and Share Intel Related to the Worldwide Threat of Sea Mines Maintain Mobility 1.0 of Operational Forces Against the Threat of Sea Mines Protect Operating 0.9 Forces Against the Threat of Sea Mines in the Littoral Preserve Freedom 0.6 of the Seas for Commercial Shipping in the Face of Sea Mines Anchored Scale for Importance of Operational Task: 3 Points--Task is critical in achieving a strategic requirement. A potential war stopper if not done. 2 Points--Task is essential in achieving a strategic task. Significantly mitigates risk. 1 Point--Task is useful and enhancing. 0 Points--Task not performed.