Using AFS021: the problem is big, time is short, and visibility is enormous.

The foundation of the Comprehensive Assessment of Nuclear Sustainment (CANS) analysis was the aggressive use of Air Force Smart Operations for the 21st Century (AFSO21) tools to attack root causes. Though the effort was time constrained and many of the processes were modified to streamline the application, this did not detract from the effort, and actually enhanced the team's ability to use those portions of AFSO21 that made sense. Overall, the CANS effort highlights the power, flexibility, applicability, and simplicity of the AFSO21 toolkit and is a resounding success story.

Introduction

When initially assigned to the Air Force CANS project, I wondered what role analysis would play in the effort. Typically, analysts are brought into projects after all the data has been collected and it is time to analyze. Most often, this is much too late for the analytic effort to have the optimum impact on the problem and its solutions. However, in this case, the CANS chairman brought me on board at the very beginning. This was a chance to shape the effort and to ensure that a methodical and repeatable analytic process was both followed and documented.

Given this phenomenal opportunity and the fact that I am an operations research analyst by trade, not an AFSO21 expert, why did I choose to use the tools of AFSO21 ? The simple answer is that it just made sense. When researching applicable industry methods for root cause analysis and risk analysis, the methods that I found most used by industry were available in the AFS021 Playbook. Additionally, because the AFSO21 process is tailorable, we were able to use an industry accepted process and tools while still meeting a very short schedule. The remainder of this article reviews the methodology used in the CANS project.

[ILLUSTRATION OMITTED]

CANS Methodology

The focus of the CANS methodology was to not only investigate nuclear sustainment and develop solutions, but also to ensure a clear linkage would exist amongst the prioritized findings, root causes, and actionable solutions for implementation.

A team of subject matter experts (SME) was selected, divided into seven subteams, and subsequently consolidated into five working teams as follows:

* Organizational structure and lines of authority and responsibility

* Logistics and supply chain management

* Maintenance and storage

* Training and standardization

* Previous report review and research

In order to ensure that the CANS study produced solutions that addressed the root causes of the problem instead of only treating the symptoms, the team followed a methodical, industry and Air Force accepted, appropriately modified, 5-step problem solving approach called Define, Measure, Analyze, Improve, and Control (DMAIC) (11) which worked as a framework, encapsulating the overall solution methodology (see Figure 1). (Please note that at the time of this study, the Air Force had not yet fully adopted the Toyota 8-step problem solving model as the preferred model for AFSO21. For more information, see the AFSO21 Web site.)

Define

The first step of the DMAIC model is to define the problem and develop an improvement project plan.

In this stage, the CANS team built subteam-level charters, defined the scope, and established milestones and roles. Additionally, based on the defined scope, the team developed a comprehensive questionnaire for the team to use during all site visits.

The overall problem was defined and scoped. From the definition, using affinity diagramming, cause and effect diagramming, and brainstorming, (3,4,5,10,11,12) the team determined and stratified key mission elements, or focus areas, contributing to the overall problem. These key mission elements are noted as follows:

* Training. Activities addressing the level of competence to execute the required job. They include formal training, education, on-the-job training, certifications, and experience.

* Policy. Activities that define how the Air Force does business. They should be clear, concise, standard, and relevant.

* Culture. Intangibles such as trust, support, accountability, internal and external environment, spirit, politics, pride, personal commitment, perceptions, and tribe mentality.

* Resources. People, equipment, systems, facilities, funding, and time.

* Oversight and Control. Activities that provide feedback on Air Force processes. They include performance measurements and metrics, inspections, closed loop feedback processes, and corrective actions.

[FIGURE 1 OMITTED] (5)

Also during this step, the research subteam collected and reviewed over 2,000 documents related to the Air Force nuclear enterprise. From this group of documents, the research team identified 67 key documents and scrutinized previous findings as they related to the key mission areas. It is important to note that the other subteam members were not given access to the previous documents so that the data collection in the site visits would not be biased.

Measure

The second step of the DMAIC model is to measure the existing process and identify the process capability requirement.

The teams collected data through a variety of methods during the measurement step. These methods include the following:

* Site visits consisting of 23 members of the team visiting 31 sites with nuclear capability or related functions

* Personal interviews during site visits, and followup interviews as needed with SMEs

* Research included staff studies, reports, policy, audits, and other sources

* A rapid improvement event addressing the engineering technical support process

Analyze

The process is analyzed to determine its capability. Data is analyzed to identify opportunities for improvement and to develop plans for improving the process. The steps in this phase include root cause analysis, solution development, risk analysis and mitigation, and determining the path forward.

Root Cause Analysis

Root cause analysis was conducted using proven methods, accepted by both industry and the Air Force. Specific methods used included flow diagramming (value stream or process), affinity diagramming, brainstorming, cause and effect diagramming, and the Five Whys. (3, 4, 5, 10,11,12) Brief descriptions of these methods follow.

* Flow Diagramming (Value Stream or Process Mapping). Value stream mapping (VSM) is a tool to visualize an entire process, such as the flow of material and information as a product or service makes its way through the value stream. It is a good method for displaying relationships between material and information, making waste and its sources visible, setting a common language and basis for discussion, and getting the big picture. Value stream mapping differs from process mapping in that it is broader in scope, tends to be at a higher level, and is typically used to identify where future focus should occur. The process map shows a process in more detail than a VSM. Such information is useful in analyzing all aspects of a specific process. VSM was used by the engineering team to map out the technical order 0025-107 maintenance assistance engineering process. Process mapping was used by the engineering team to map out the information flow of the time change technical order process. The CANS team did not perform a full VSM on the entire Air Force nuclear sustainment enterprise due to time constraints. However, the team did use the tool to visualize the highest-level processes of the entire enterprise in order to scope the problem and to view the entire enterprise as one overall process. This was helpful as it highlighted the seams to organizations outside of the Air Force and was especially useful in integrating process solutions to non-Air Force processes.

* Affinity Diagramming. Affinity diagramming, sometimes called the JK Method for its creator Jiro Kawakito, is useful for organizing and presenting large amounts of data (ideas, issues, solutions, problems) into logical categories based on user perceived relationships and conceptual frameworks. When paired with brainstorming, affinity diagrams can help organize data and ideas, group like items, sort a large number of brainstorming ideas quickly, build consensus, avoid long discussions, stop people from dominating discussions, stimulate independent thoughts, and enable a greater variety of ideas. The CANS team used affinity diagramming when determining the five key mission areas.

* Brainstorming. Brainstorming is a problem solving technique in which team members attempt a deductive methodology for identifying possible causes of any problem via free-form, fast-paced idea generation. Brainstorming was popularized by Alex Osborn (advertising executive) in the 1930s, and can be an effective means to develop many ideas in a short amount of time. Brainstorming was used throughout the CANS study.

* Cause-Effect Diagramming (Fishbone Diagramming). Cause-effect diagramming, also called fishbone or Ishikawa diagramming, was created by Kaoru Ishikawa in the 1960s as part of the quality movement at Kawasaki Shipyards. It is a visual tool used to logically organize possible causes for a specific problem or effect by graphically displaying them in increasing detail. Additionally, it helps to identify root causes and ensures common understanding of the causes. In this method, a problem statement is written in a box on the right side of the diagram and then possible causes are determined (usually via brainstorming) as categories branching off the problem statement. Benefits include conciseness, adding structure to brainstorming, easily trained and understood, works well in team environment, and the ability to determine and analyze countermeasures. This method was used in determining the five key mission areas and during root cause analysis.

* The Five Whys. For root cause analysis, the team used the Five Whys, a well accepted method, first developed by Sakichi Toyoda of Toyota, described by Taiichi Ohno as "... the basis of Toyota's scientific approach," and is now widely used across industry and within AFSO21. The Five Whys typically refers to the practice of asking, five times, why the failure has occurred in order to get to the root cause or causes of the problem. There can be more than one cause to a problem as well. In an organizational context, generally root cause analysis is carried out by a team of persons related to the problem. No special technique is required.

Using these tools, the hundreds of tactical findings discovered during data collection were analyzed to determine common trends or higher-level issues, which the team chose to call strategic level findings. These findings were then analyzed to determine the root causes. Finally, solutions were developed and then further scrutinized via a murder board process to ensure they truly solved the root causes instead of merely symptoms of the real problem.

Risk Analysis

Risk analysis (2,14) and mitigation was performed on each solution using a modified version of the Develop and Sustain Warfighting Systems (D&SWS) Core Process Working Group (13) Active Risk Management (ARM) Process model. Because of the high visibility and importance associated with the correction of the enterprise, the risks of not implementing the solutions were assumed to be known and sufficiently high such that all solutions would be implemented. Thus, the risk analysis in this study focused on the risks associated with implementing the solutions.

These risks were identified and analyzed as follows. The teams identified potential risks to solutions via brainstorming with SMEs by indentifying and explicitly defining potential unintended consequences which might occur when the solutions are implemented. These consequences were then scored by the SMEs, via a Delphi voting method, using life cycle risk management likelihood and severity ratings as defined in the D&SWS ARM Process model and shown in Tables 1 and 2. (Note that the CANS team focused on performance impact as the most critical characteristic. Each proposed solution was reviewed on the basis of consequence, vice cost or time to implement.)

Notional risk analysis output is shown in Figure 2, where the green squares identify a safe area where there is little likelihood of a risk occurring and low impact to the system if it does. Similarly, the yellow and red squares identify medium and high risk areas, respectively. The line is calculated by measuring the full range of the yellow area (medium impact) and determining the 98 percentile point. The team determined that the +98 percentile data points (within the medium area), could have very easily been scored within the red area (high impact) relative to the error margins within the scoring process and should be treated as high risk. Thus, solutions with risks above and to the fight of this line required additional review by the teams to determine risk mitigation strategies.

Prioritization via Multi-Objective Optimization

To determine a prioritized order, the strategic level findings were scored on their impact, if solved, on the five key mission areas. The result was then modeled as a multi-objective optimization problem in which five key mission areas represent the competing objectives and the prioritized order of the strategic findings represents the decision variable. In this type of problem, there often exists no single criterion for choosing the best solution. In fact, even the notion of best can be unclear when multiple objectives are present; and in many cases, it can be shown that improvement to one objective actually degrades the performance of another. (1)

The multi-objective optimization problem,

min F(x)

subject to

x [epsilon] [OMEGA] = [{0,1.).sup.n]: [g.sub.i] (x)<O, i = 1,2,..., M}

where F: {0,1}" [right arrow] RJ is that of finding a solution [x.sub.n] [epsilon][OMEGA] that optimizes the set of objectives F = ([F.sub.1], [F.sub.2],.... [F.sub.j]) in the sense that no other point y [epsilon] [OMEGA] yields a better function value in all the objectives. (15) (Note the precise mathematical definition of [x.sub.n] can be found in Ehrgott (8)) The point x is said to be nondominated, efficient, or optimal in the Pareto sense. (9) The (typically infinite) set of all such points is referred to as the Pareto optimal set or simply the Pareto set. The image of the Pareto set is referred to as the Pareto Frontier or Pareto Front. If the Pareto set (or corresponding Pareto front) results from a solution algorithm and is not exact, it is referred to as the approximate (or experimental) Pareto set or approximate (or experimental) Pareto front, respectively.

Once defined, a multi-objective optimization problem can be solved via many methods. The particular method selected can depend on many factors including, but not limited to, the complexity of the problem, the time allowed for problem solution, the availability and quality of information, and the preferences of t h e decisionmaker. In this case, an a priori scalar method called weighted-sum-of-the-objective-functions (WSOTOF) was selected. As the name implies, this method combines the various objectives via a convex combination (a weighted sum). Though it is among the simplest of the multi-objective methods, it is guaranteed to produce an efficient solution (see Lemma 3.3.11 in Walston (19)). It should be noted that this method is not guaranteed to find all possible solutions, particularly if the corresponding Pareto front is non-convex; (6,7,16,17) however, in this particular case, the benefits of simplicity and speed far outweigh potential risks associated with examining only a portion of the Pareto front.

[FIGURE 2 OMITTED]

To combine the objectives, the WSOTOF method requires a predetermined set of weights. In many cases, this can be problematic (18) as it is dependent on subjective judgment of the decisionmaker which may not be available or fixed across the duration of the study. Thus, this step is of particular importance. Additionally, in this particular problem, the determination of weights is even more complex as there are multiple decisionmakers to be considered.

To ensure that multiple decisionmaker preferences are included and considered in the solution, the following method was used. First, a group of senior Air Force leaders was identified as stakeholders for the nuclear sustainment enterprise and defined as the decisionmakers for the multi-objective problem. After each stakeholder provided a set of weights, the problem was solved as follows:

* A simple average of the weights provided by the stakeholders was used as the weights for the problem. However, there was considerable variance in the weighting schemes provided by the stakeholders (see Figure 3 and Table 3) indicating that further investigation was necessary. The distribution of the weights was tested for normality using normal p-p plots and the Kolmogorov-Smirnov (K-S) goodness test for normality. The plots and the K-S test indicate failing to reject the null hypothesis that the weights are normally distributed. Though in this case, parametric statistics would then be applicable, the use of a simple mean may not be adequate because of the high degree of variance.

* The weights were further analyzed as follows. A sensitivity analysis was conducted to determine the impact of the weighting scheme on the overall prioritized solution. It was found that the top priority issues in the prioritization solution were relatively impervious to the weighting scheme. A prioritized list of findings was determined for each decisionmaker' s preference of weights and was then examined against the others. In this case, it was also found that the top priority issues did not vary much over the various weighting schemes. The average of the ranks assigned from each weighting scheme was determined for each finding, and was used to assign its final rank.

[FIGURE 3 OMITTED]

Once the objectives have been combined, any applicable optimization method can be used to determine the prioritized list of findings. In this case, because no constraining information was identified, and impact to the overall problem statement was the sole criteria for selection, a simple greedy heuristic method was used. Simply stated, once the weights are determined, the value of solving each particular finding becomes clear, and the prioritized list follows directly.

Cost Analysis

The CANS cost team estimated costs for solutions that required funding. Cost analyst support upfront was critical to providing leadership with vital financial information. As solutions were identified, the cost team worked to define tasks, time lines, and associated costs. Identifying and linking costs with solutions allows leadership to make timely, informed decisions with known costs. In this case, costs of the CANS solutions totalled $25.6M for fiscal year 2008--the process worked and our leadership provided the funding to fix the problems because the methodology was solid.

Improve. During the Improve step, the plan that was developed in the Analyze phase is implemented. The results of the change are evaluated and conclusions are drawn as to its effectiveness. This can lead to documenting changes and updating new instructions and procedures.

The CANS chairman was given authority to immediately implement some solutions. There were six just-do-it solutions. The remaining results of this team's efforts were presented to senior leaders in a number of briefings at the major commands and Air Staff.

Control. Control plans were developed to ensure the process is institutionalized and continues to be measured and evaluated. This can include implementing process audit plans, data collection plans, and plans of action for out-of-control conditions, if they occur.

This study team worked concurrently with SAF/IG (Secretary of the Air Force, Inspector General' s office) and AF/A9 (Studies and Analyses, Assessments, and Lessons Learned Directorate) to develop inspection and assessment criteria and plans to assess the status of the Air Force nuclear sustainment enterprise and measure the progress of addressing the CANS findings.

Conclusion

The foundation of the CANS analysis was the aggressive use of AFSO21 tools to attack root causes. Though the effort was time constrained and many of the processes were modified to streamline the application, this did not detract from the effort, and actually enhanced the team's ability to use those portions of AFSO21 that made sense. Overall, the CANS effort highlights the power, flexibility, applicability, and simplicity of the AFSO21 toolkit and is a resounding success story.

Notes

(1.) A. Abraham and J. Lakhmi, J. Evolutionary Multi-objective Optimization, London: Springer-Verlag, 2005.

(2.) AFMC PAM 63-101, Risk Management, 9 July 1997.

(3.) AFSO21 Level 1 Training, Lesson 4, Value Stream Mapping, accessed https://wwwd.my.af.mil/afknprodlASPs/deskbook/ index2.asp?Filter=OO-21&Type=DP&Cat=21, 27 June 2008.

(4.) AFSO21 Level 1 Training, Lesson 7, Problem Solving Tools, [Online] Available: https://wwwd.my.af.mil/afknprod/ ASPsldeskbooklindex2.asp?Filter=OO-21&Type=DP&Cat=21, 27 June 2008.

(5.) AF/SAO, Air Force Smart Operations for the 21st Century Playbook, Version 2.0, Volume H, H-2, [Online] Available: https://wwwd.my.af.mil/afknprod/ASPs/docman/ DOCMain.asp?Tab=0&FolderID=OO-TR-AF-43-13&Filter=OO-TRAF-43, October 2007.

(6.) Y. Collette, and P. Siarry, Multiobjective Optimization Principles and Case Studies, Berlin, Germany: Springer-Verlag, 2004.

(7.) I. Das and J. D. Dennis, Jr, "A Closer Look at Drawbacks of Minimizing Weighted Sums of Objectives for Pareto Set Generation in Multicriteria Optimization Probems, Structural Optimization, 1997, 14, 63--69.

(8.) M. Ehrgott, Multicriteria Optimization, second edition, Berlin, Germany: Springer, 2005.

(9.) Y. Fu and U. Diwekar, "An Efficient Sampling Approach to Multiobjective Optimization," Annals of Operations Research, 132, 109-134, 2004.

(10.) M.L. George, D. Rowlands, M. Price, and J. Maxey, Lean Six Sigma Pocket Toolbook, a Quick Reference Guide to Nearly 100 Tools for Improving Process Quality, Speed, and Complexity, New York, NY: McGraw-Hill, 2005.

(11.) T. Goldsby and R. Martichenko, Lean Six Sigma Logistics, Strategic Development to Operational Success, Fort Lauderdale, Florida: J. Ross Publishing, Inc., 2005.

(12.) SixSigma Magazine, term definition library, accessed http:// www.isixsigma.com/dictionary/glosary.asp, 27 June 2008.

(13.) J. Jackson and M. Hailstone, "D&SWS Life Cycle Risk Management (LCRM) Update," briefing, D&SWS Core Process Working Group, 27 Jun 2008.

(14.) J. Kindinger and J. Darby, "Risk Factor Analysis--A New Qualitative Risk Management Tool," Proceedings of the Project Management Institute Annual Seminars & Symposium, Houston, Texas, 7-16 September 2000.

(15.) L. H. Lee, E. P. Chew, and S. Teng, "Optimal computing budget allocation for multi-objective simulation models," in J.S.S.R.G. Ingalls, M. D. Rossetti and B. A. Peters, editors, Proceedings of the 2004 Winter Simulation Conference, [Online] Available: http://www.informssim.org/wsc04papers/071.pdf.

(16.) M. Makowski, "Methodology and a Modular Tool for Multiple Criteria Analysis of 1p Models," working papers, Laxenburg, Austria: International Institute for Applied Systems Analysis, 94-102.

(17.) S. Sait and H. Youssef, lterative Computer Algorithms with Applications in Engineering: Solving Combinatorial Optimization Problems, Las Alamitos, California: IEEE Computer Society, 1999.

(18.) P. Vincke, Multicriteria Decision-Aid, England:John Wiley & Sons, 1992.

(19.) J. Walston, Search Techniques for Multi-Objective Optimization of Mixed Variable Systems Having Stochastic Responses, PhD thesis, Air Force Institute of Technology, September 2007.

Major Jennifer G. Walston, PhD, is currently the Chief Scientist, Air Force Logistics Management Agency, Maxwell AFB, Gunter Annex, Alabama. At the time of writing this article, she was temporarily assigned to the Comprehensive Assessment of Nuclear Sustainment project at Hill Air Force Base, Utah.

Major Jennifer G. Walston, PhD, USAF

Table 1. Consequence Likelihood Ratings (13)

1   Not Likely         1%-20%
2   Low Likelihood   21 %-40%
3   Likely           41 %-60%
4   Highly Likely    61 %-80%
5   Near Certainty   81 %-99%

Table 2. Risks

               DoD Guide                Proposed Air Force Definition

1   Minimal or no consequence to       Minimal consequence to technical
    technical performance              performance but no overall
                                       impact to the program success. A
                                       successful outcome is not
                                       dependent on this issue; the
                                       technical performance goals will
                                       still be met.

2   Minor reduction in technical       Minor reduction in technical
    performance or supportability,     performance or supportability,
    can be tolerated with little or    can be tolerated with little
    no impact on program               impact on program success.
                                       Technical performance will be
                                       below the goal, but within
                                       acceptable limits.

3   Moderate reduction in technical    Moderate shortfall in technical
    performance or supportability      performance or supportability
    with limited impact on program     with limited impact on program
    objectives.                        success. Technical performance
                                       will be below the goal, but
                                       approaching unacceptable limits.

4   Significant degradation in         Significant degradation in
    technical performance or major     technical performance or major
    shortfall in supportability; may   shortfall in supportability with
    jeopardize program success.        a moderate impact on program
                                       success. Technical performance
                                       is unacceptably below the goal.

5   Severe degradation in technical    Severe degradation in
    performance; cannot meet KPP or    technical/supportability
    key technical/supportability       threshold performance; will
    threshold; will jeopardize         jeopardize program success.
    program success

Table 3. Descriptive Statistics

                         N         Minimum      Maximum

Training                 31           5            40
Policy                   31          10            50
Culture                  31           5            35
Resources                31           5            40
Oversight Control        31           5            30
Valid N (listwise)       31

                                   Standard
                        Mean      Deviation     Variance

Training               22.16        7.267        52.806
Policy                 21.77        8.995        80.914
Culture                16.06        8.668        75.129
Resources              22.52        8.282        68.591
Oversight Control      17.48        5.591        31.258
Valid N (listwise)