Printer Friendly

Scandal and tragedy? Or acquisition lessons relearned by the F-35 Program.

The U.S. Congress has long been concerned about controlling schedule and cost overruns, and attaining expected performance in major defense acquisition programs (MDAP; Blickstein, Nemfakos, & Sollinger, 2013). Schedule, cost, and performance are the three inextricably linked pillars of acquisition (Defense Acquisition University [DAU], 2006).

The 1986 President's Blue Ribbon Commission on Defense Management, referred to as the Packard Commission Report, determined that MDAPs take too long to develop, cost too much, and incorporate obsolete technology by the time they are fielded. More recently, the 2009 Weapon Systems Acquisition Reform Act sought to improve the likelihood of success of MDAPs by focusing on decisions and types of strategies at their inception K (Eide & Allen, 2012; Young et al., 2010). Out of 95 known MDAPs from 2006 to 2010, 40% experienced delays of up to 2 years (Young et al., 2010). Total MDAP cost overruns have averaged between 20 and 54% (Melese, Franck, Angelis, & Dillard, 2007). In 2011, the DoD's portfolio of 96 MDAPs stood at $1.58 trillion--$74.4 billion or 5% more than in 2010 (Government Accountability Office [GAO], 2012a). The F-35 Joint Strike Fighter (JSF) program, which seeks to develop and procure 2,457 aircraft for the United States, is the largest single global defense program in history at $386 billion. It accounted for $10 billion, or 13%, of the cost growth in 2011 (GAO, 2011b). Recently, Senator John McCain, chairman of the Senate Armed Services Committee, said the F-35 program's record and performance "has been a scandal and a tragedy" (Associated Press, 2016).

Like previous joint aircraft programs (F-111, V-22, and T-6), the F-35 JSF program has been subject to schedule, cost, and performance shortcomings due to acquisition strategy challenges identified by U.S. Government agencies and U.S. Government think tanks (Blickstein et al., 2013; GAO, 2012c). These acquisition strategy challenges include Balancing Requirements, Harnessing Technology, Demanding Commonality, Evoking Concurrency, and Encouraging Partnering (Blickstein et al., 2013; Dunne, 2011; Ergas, 2009; GAO, 2012b; Wicht & Crawley, 2012). This article summarizes a 2015 dissertation's research methodology and lessons learned about strategy challenges from a triangulated, qualitative case study analysis on previous joint aircraft acquisition programs, on governmental agency and think tank reports and scholarly journals on the JSF, and on questionnaire responses by subject matter experts (SME) who were currently working on the JSF program. Secretary of Defense (SecDef) Robert Gates (2009) complained that acquisition problems have been persistent and difficult despite congressional involvement in trying to resolve them (more than 125 studies since World War II have produced no comprehensive, effective, and permanent solutions). The goal of this unique research is to improve the cost, schedule, and performance of the JSF program and other MDAPs by understanding better, and making recommendations on, the acquisition strategy challenges of Balancing Requirements, Harnessing Technology, Demanding Commonality, Evoking Concurrency, and Encouraging Partnering. The research shows the F-35 program relearned some old lessons and learned some new ones.

The F-35 is a stealthy, supersonic, multirole fighter built by Lockheed Martin (LM) in three variants to penetrate modern integrated air defenses. The U.S. Air Force (USAF), several Partners, and Foreign Military Sales (FMS) countries will fly the F-35A Conventional Take-Off and Landing (CTOL). The U.S. Marine Corps (USMC), a couple of Partners, and a possible FMS country will fly the F-35B Short Take-Off/Vertical Landing (STOVL). The STOVL is the same size as the CTOL, but the STOVL carries less fuel because of the vertical lift fan. The U.S. Navy (USN) and the USMC will fly the F-35C Carrier Variant with a much larger wing than the other models for carrier landing approach speeds, but it is not being internationally marketed.

Methodology

A triangulated, multilayered, qualitative case study was used to synthesize lessons from three lanes of analysis: the comparison of previous joint aircraft acquisition programs (F-111, V-22, and T-6), the mining and coding of government agency and think tank reports and scholarly journals on the F-35 program, and the mining and coding of questionnaire responses from SMEs who were currently working on the F-35 program at the time of the research in 2015. The main research question of this study was to understand how acquisition strategy challenges (Harnessing Technology, Demanding Commonality, Evoking Concurrency, and Encouraging Partnering) have both helped and hindered joint aircraft programs' schedule, cost, and performance (the pillars) in terms of the acquisition strategy's original intent, its positive and negative effects on the pillars, and what improvements could be made.

Jogulu and Pansiri (2011) supported triangulation over a single approach because it strengthens findings and inferences made for understanding program management discipline. Yin (1994) and Patton (2002) believed a deep-rooted and multilevel case study analysis could help formulate appropriate relationships between phenomena. As for mining and coding, Patton (2002) offered how to use qualitative data analysis (QDA) software to categorize and to make sense out of massive amounts of data. Finally, submitting a questionnaire with phenomenological attributes to a heuristic group of SMEs produced high-fidelity qualitative analysis of the experiences, beliefs, and perceptions of respondents from multiple perspectives (Moustakas, 1990; Shank, 2006). The questionnaire is phenomenological because the respondents answer the questions as they perceive the situation (Moustakas, 1990). This approach is heuristic because the research process involved the experiences of the researcher in relation to the questionnaire respondents (Moustakas, 1990). As chief of the JSF Coordination and Training Office (JCTO) at the USAF's Air Education and Training Command (AETC) for 4 years, the researcher constantly reflected on and interpreted daily interactions with SMEs from the F-35 Joint Program Office (JPO), LM, and the USAF's headquarters, test community, and training operators. Wacker (1998) concluded that, when dealing with the social sciences (including program management), good recommendations come from open questions, often applying the researcher's and SME's own experiences, instead of from scientific, quantifiable analysis.

Each analysis lane is considered multilayered because three previous joint aircraft acquisition programs were reviewed and because three types of documents and three categories of questionnaires were mined and coded. The F-111, V-22, and T-6 programs were the most suitable comparisons in the number of Services initially interested in combining development, the number of aircraft being procured, and the overall complexity. Potential sampling of previous joint aircraft acquisition programs could have included some well-known and successful aircraft acquisition programs like the F-4, F-5, and A-7 that were used by the Air Force, Navy, and Marines, and several other countries, but they were originally developed by a single Service first (Antill & Ito, 2013; Pike, 2011). Although scholarly journals and think tank reports exist on the F-35 program, General Accounting Office/GAO and Director of Operational Test and Evaluation (DOT&E) reports on the F-35 outnumbered the scholarly journals and think tank reports by 44 to 24 in this study. General Accounting Office/GAO and DOT&E reports on the JSF go back to 2003, when some of the acquisition strategy challenges began to emerge and provided vast amounts of numerical and statistical data, as well as well-defined problems and recommendations. The General Accounting Office/GAO and DOT&E reports on the JSF were prepared by experts in the field of acquisition; since the reports were a matter of public record, this promoted validity and dependability. The original plan was to e-mail the questionnaire to about 50 JSF SMEs in three categories--20 out of 200 from the F-35 JPO as the managers, 10 out of 100 from LM as the providers, and 20 out of 200 from the U.S. military services as the customers. Leedy and Ormrod (2009) recommended an unstructured survey to sample between five and 25 individuals. The 2015 study accepted the upper limit of 25 within the manager and customer categories.

The disclosure that the researcher was AETC's chief of the JCTO during the time of the study promoted credibility and integrity with the questionnaire participants, besides contributing to a heuristic research approach. As for construct validity on the unstructured questionnaire, the 2015 study followed the approach from a questionnaire in a dissertation by Uda (2012). The questionnaire in the 2015 study was championed by the JPO deputy program executive officer and vetted by the JPO security officer and lawyers. A limitation of the 2015 study was that only unclassified information was used from open source literature and unclassified answers from the questionnaire. One of the toughest challenges to internal validity was the need to guard against the researcher's and respondents' expectancies and biases while being so intimately involved with the F-35 program. The participants were not led to foregone conclusions through an interview; this is why an open questionnaire was used instead. The 2015 study was delimited in scope by using a qualitative methodology instead of a quantitative one, because the large amount of government data would have been difficult to quantify any actionable recommendations. DoD instructions and university institutional review board processes ensured ethical standards were maintained with respect to questionnaire participants to the point where each participant's commander or supervisor approved contact by the researcher.

Previous Joint Aircraft Acquisition Programs

Three previous joint aircraft programs similar in scope to the JSF were compared: the 1960s Tactical Fighter Experimental (TFX) F-111, the 1980s Joint Service Vertical Takeoff & Landing Experimental (JVX) V-22, and the 1990s Joint Primary Aircraft Training System (JPATS) T-6 programs. Examining the effect on the pillars of acquisition by the previous programs' acquisition strategy challenges acted as a precursor to what the JSF program has experienced. As previously discussed, although there were successful aircraft acquisition programs like the F-4, F-5, and A-7, they were developed by a single Service (Antill & Ito, 2013; Pike, 2011). There are also some successful joint missile programs like the Joint Direct Attack Munition, but it simply did not match the scope, scale, and complexities of the F-111, V-22, and T-6 programs.

Tactical Fighter Experimental (TFX) F-111 Program

The TFX program introduced the multimission concept that would affect the attack aircraft industry for the next few decades (Miller, 1982). After World War II, attack aircraft were developed for single purpose missions: nuclear strategic bombing, tactical interdiction, air superiority, or close air support (Miller, 1982). Furthermore, SecDef McNamara wanted to shift doctrine from massive nuclear retaliation to a range of conventional options (Coulam, 1977). Still, the USAF wanted a follow-on F-105 fighter-bomber for the delivery of internally carried tactical nuclear missiles (TFX Contract Investigation, 1963). The USN wanted an air-to-air missile carrier to identify and shoot down enemy planes at extended ranges from their carriers (Coulam, 1977). The USAF and the USN could agree only on a swing-wing, two-seat, and twin-engine design (Art, 1969). The USAF wanted a tandem-seat aircraft (pilot in front and weapon system operator behind) for low-level penetration ground-attack, while the USN wanted a shorter, high-altitude interceptor with side-by-side seating to allow the pilot and radar intercept officer to share the radar display (Miller, 1982). Coulam (1977) concluded that directly competing requirements were inevitably traded off, never fully meeting either Service's requirements.

The TFX program resulted in the F-111, produced by General Dynamics (GD), serving primarily as a supersonic, medium-range interdictor and tactical attack aircraft that later filled the roles of strategic bomber and electronic-warfare aircraft (Logan, 1998). It first entered service with the USAF in 1967 and then with the Royal Australian Air Force in 1973 (Logan, 1998). The F-111 featured new variable-geometry wings for high- and low-speed flight with leading-edge slats and double-slotted flaps over its full length to create more lift for relatively short runway use. It also had afterburning turbofan engines and automated terrain-following radar for low-level, high-speed flight (Logan, 1998). A major failing of the TFX program was that it asked too much of technology too soon (TFX Contract Investigation, 1970). SecDef McNamara and the Services looked to GD to solve all of the issues with new innovations (TFX Contract Investigation, 1963). Although variable-geometry wings worked as advertised, the poor performance of the Mark II low-level avionics, T-30 turbofan engines, and variable inlets would plague the military services (TFX Contract Investigation, 1970). The program also experienced a 25% concurrency rate--141 out of 547 total USAF F-111s needed retrofits (Coulam, 1977; Richey, 2005).

Cost estimates also concerned SecDef McNamara (TFX Contract Investigation, 1970). He recognized that the Services had limited resources and funding from Congress, so in order to get more from their budgets, they would encourage bids that were unrealistically low (TFX Contract Investigation, 1963). Once the Services had congressional support and dedication to continue the program, costs predictably rose, but the Services were likely to get additional funds to finish the program (TFX Contract Investigation, 1963). Most USAF programs in the 1950s exceeded their costs by 100 to 200% and their schedules by 36 to 50% (Summers, 1965). McNamara passionately drove for a single aircraft to meet the needs of both the USAF and USN, expecting a high degree of commonality between the two versions (TFX Contract Investigation, 1963). GD planned to reduce cost and risk by adding Grumman and P&W as aircraft development partners (TFX Contract Investigation, 1963). GD was also able to reduce predicted unit price by courting the U.K. early as a partner to buy several of a specialized variant of the F-111, expecting commonality to save money (Hunter, 1998; Logan, 1998). SecDef McNamara chose GD over Boeing for more realistic cost estimates, but the Services were guilty of assuming high expectations on technical performance (TFX Contract Investigation, 1970). In the end, the TFX program had a 100% cost overrun and a 30 to 40% schedule overrun (TFX Contract Investigation, 1970), and only seven F-111Bs were built for the USN for test purposes before they cancelled out of the program (General Accounting Office, 1973).

Joint Service Vertical Takeoff and Landing Experimental (JVX) V-22 Program

The V-22 is a multimission, tilt-rotor aircraft with both a vertical takeoff and landing (VTOL) capability like a helicopter and a fixed-wing aircraft capability, achieved by tilting its wing-mounted rotors to act as propellers (General Accounting Office, 1990, 1994; Whittle, 2010). The JVX program started in 1981 to meet joint Service requirements that would satisfy USMC medium-lift assault, USN search and rescue, and USAF long-range special operations (General Accounting Office, 1990, 1994, 1997). The DoD awarded Bell Helicopter and Boeing Helicopters a development contract in 1983 (Whittle, 2010). The U.S. Army (USA) planned to use the USMC's assault requirements for its medium cargo lift and medical evacuation needs (General Accounting Office, 1986). When the first V-22 rolled out in 1988, the USA had already left the program for good to focus its budget on more immediate aviation programs (Whittle, 2010).

The JVX's original program cost estimates changed significantly, and its development process was long and controversial (Whittle, 2010). One of the USN's cost-saving strategies for the USMC's MV-22s included a high level of concurrent development (General Accounting Office, 1994). The General Accounting Office (1994) warned that such concurrency involved high risk that eventually required rescheduling and spending on increased overtime. The V-22 began flight testing in 1989 and started design alterations immediately (Whittle, 2010). The complexity and difficulties of being the first tilt-rotor intended for military service in the world led to many years of development (Whittle, 2010). The JVX program faced opposition in the Senate in 1989, surviving two separate motions that both could have resulted in program cancellation (Whittle, 2010). The full-scale development contract was even terminated once in October 1992 because Bell and Boeing failed to assemble all six flight-test aircraft, failed to perform all planned drop and fatigue tests, and did not complete all flight testing (General Accounting Office, 1994).

The V-22 program entered full-rate production without mature manufacturing processes that required a redesign and retrofit of the hydraulic and electric system and led to a Nunn-McCurdy breach in 2001 (General Accounting Office, 2003). Although the USMC began crew training for the MV-22 Osprey in 2000, it did not declare initial operational capability (IOC) until 2007 (Whittle, 2010). The Osprey's other current operator, the USAF, declared IOC in 2009 with their CV-22 version of the tilt-rotor (Whittle, 2010). Although 12 MV-22s deployed to Iraq in January 2009, and confirmed there that the MV-22's enhanced speed and range enabled personnel and internally carried cargo to be transported faster and farther than by legacy helicopters (GAO, 2009), almost 30 years had passed from program inception to real-world execution. During that time, V-22 costs have risen sharply above initial projections--1986 estimates (stated in fiscal year 2009 dollars) that the program would build nearly 1,000 aircraft in 10 years at $37.7 million each have shifted to fewer than 500 aircraft at $93.4 million each--a procurement unit cost increase of 148%, while research, development, testing, and evaluation costs increased over 200% (Gertler, 2011). Even after the Department of State approved Japan in 2015 for acquisition of up to 17 V-22B Block-C Ospreys and all the logistical support, Japan deferred their purchase indefinitely due to their budget restraints and the predicted increased costs to maintain the complicated weapon system (McCullough, 2015).

Joint Primary Aircraft Training System (JPATS) T-6 Program

In 1988, the USAF and the USN worked together on the DoD Trainer Aircraft Masterplan and formed the JPATS program to modernize their training aircraft fleets and methods of primary flight training (AETC, 2010). Once the USAF and USN finally agreed on tandem cockpits (the instructor behind the student) and the anthropometrics of the ejection seat to allow more women into flight training, the USAF and USN settled on the commercial-off-the-shelf (COTS) Hawker Beechcraft (formerly Raytheon Aircraft Company) Pilatus PC-9 aircraft (AETC, 2010).

Military-unique design requirements off the COTS baseline grew from about 5% when the program entered limited production in 1995 to almost 70% by the early 2000s (General Accounting Office, 2003). Furthermore, balancing requirements between the USAF and USN led to a 22% heavier aircraft than its original COTS version (Gantt, 2002). It took 7 years from the establishment of the DoD Trainer Aircraft Masterplan and JPATS program in 1988 to aircraft coming off the assembly line in 1995 for a relatively simple mission (AETC, 2010). Twelve years into production in 2007, JPATS experienced a "critical" Nunn-McCurdy breach, exceeding 50% cost growth from its baseline (GAO, 2007). A DoD review concluded that the cost growth was attributed to changes in government requirements, and the remainder was due to immature and unchecked manufacturing processes (GAO, 2007). Once JPATS was rebaselined in 2008 for cost and schedule as required by Nunn-McCurdy, several foreign countries signed contracts directly with Hawker Beechcraft via direct commercial sales vice FMS (DoD, 2012).

Assessment of Previous Joint Aircraft Programs

The TFX F-111, JVX V-22, and JPATS T-6 programs experienced similar instances of acquisition strategy challenges in relation to the pillars of acquisition. Several specific instances were tabularized in the dissertation study this article is based upon. Each program had at least one acquisition strategy challenge affecting more than one acquisition pillar at the same time or one of the challenges worked in tandem with another to affect either one or more of the pillars. Figure 1 begins to answer the research subquestions concerning acquisition strategy challenges' positive and negative effects on the pillars of acquisition just by looking at the up or down arrows beside each factor below each pillar column in relation to the acquisition strategy challenge on the left. For example, at the top of Figure 1 at the intersection of "Cost" and "Balancing Requirements," a down arrow representing reduced cost stands next to the comment "Combine programs to get one aircraft," while an up arrow stands next to the comment "Meet multi-Services' missions and operating environments." The important message that Figure 1 is trying to convey is that acquisition strategy challenges are not mutually exclusive, exemplified by having the color of one strategy affecting one or more pillars of another strategy. For example, Balancing Requirements (red) and Demanding Commonality (yellow) significantly overlap and affect each other. Cost and performance are highlighted by the opposite color of the strategy challenge because it is difficult to meet the Services' individual mission needs and preferences. Likewise, Harnessing Technology (blue) and Evoking Concurrency (green) overlap, affecting mostly performance and schedule because there can never be enough engineering and modeling of new technology when that new technology is already in the production line with few actual test flights.

Government Agency and Think Tank Reports and Scholarly Journals on the JSF

Government agency and think tank reports and scholarly journals on the JSF program were mined and coded for acquisition strategy challenges and pillars of acquisition, and to see whether and how recommendations were executed. Although ATLAS.ti QDA software assisted in mining and coding, it was mostly used for cataloging and for determining strong associations between acquisition strategy challenges and pillars of acquisition, and within one another by counting co-occurrences and determining c-coefficients (Friese, 2013). Friese (2013) recommended that strong associations should be for c-coefficients [greater than or equal to] 0.08. The 2015 study addressed each strong association as long as there were two co-occurrences.

Within the "super-family" code of acquisition strategy challenges, the following coding families were established: Balancing Requirements, Harnessing Technology, Demanding Commonality, Evoking Concurrency, and Encouraging Partnering. Although the research questionnaire asked three questions on each strategy, there were really four coding components for each family code: "original intent, negative effects, positive effects, and improvement recommendations." Within ATLAS.ti, they were coded as Balancing Requirements Intent (BRI), Balancing Requirements Negative (BRN), Balancing Requirements Positive (BRP), Balancing Requirements Recommendation (BRR), Technology Intent (TI), Technology Negative (TN), Technology Positive (TP), Technology Recommendation (TR), Commonality Intent (CMI), Commonality Negative (CMN), Commonality Positive (CMP), Commonality Recommendation (CMR), Concurrency Intent (CCI), Concurrency Negative (CCN), Concurrency Positive (CCP), Concurrency Recommendation (CCR), Partnering Intent (PTI), Partnering Negative (PTN), Partnering Positive (PTP), and Partnering Recommendation (PTR).

Within the "super-family" pillars of acquisition code, the coding families of "cost, schedule, and performance" were created. After the literature review and pre-reading the respondents' questionnaires, it was appropriate to have five coding components for each pillar family code to separate the reasons from other statements. For cost, the codes were Cost Increased (CI)--bad, Cost Increase Reason (CIR)--why, Cost Decreased (CD)--good, Cost Decrease Reason (CDR)--why, and Cost Recommendation (CR). For schedule, the codes were Schedule Lengthened (SL)--bad, Schedule Lengthened Reason (SLR)--why, Schedule Shortened (SS)--good, Schedule Shortened Reason (SSR)--why, and Schedule Recommendation (SR). For performance, the codes were Performance Reduced (PRD)--bad, Performance Reduced Reason (PRR)--why, Performance Improved (PI)--good, Performance Improved Reason (PIR)--why, and Performance Recommendation (PR).

Table 1 shows the breakdown of family codes by the type of government report or scholarly journal. As previously mentioned, Government Accountability Office (GAO) and DOT&E reports on the F-35 outnumbered the scholarly journals and think tank reports by 44 to 24 in this study. Out of 68 total documents, the GAO was the largest represented group, with 33 total documents between Selected Acquisition Reports and specific reports on the F-35. Although the GAO had more documents than the scholarly journals, the GAO usually had the same researchers investigating and writing the reports for several years, so there is a reputation of expertise that could not be ignored. With only 11 documents, DOT&E was coded the most, and its highest percentage of codes went to the Harnessing Technology family of codes. DOT&E had numerous Technology Recommendation (TR) codes to adjudicate the large number of Technology Negative (TN) codes. The GAO also coded the Harnessing Technology family of codes pretty often, but it highlighted mostly TN aspects, some Technology Positive (TP) aspects, and very few Technology Recommendations (TR) as compared to DOT&E. Think tank coding and scholarly journal coding were evenly distributed among the strategies and pillars, but scholarly journals coded the Encouraging Partnering family of codes the most. It did not go unnoticed that the government agency/think tank reports coded more negatively across most strategies; whereas scholarly journals only slightly favored negative codes, but in terms of broad program management. There were 234 passages coded as recommendations from the governmental agency/think tank reports and scholarly journals out of 798 total codes.

Table 2 provides insight into the opening (O), working (W), and closing (C) timing of 80 official recommendations made by the DOT&E, the GAO, and the DoD Inspector General directly to the JSF JPO. Most of DOT&E's closed recommendations to the JPO were divided between improvements for flight test strategy, planning and realism, and with Autonomic Logistics Information System (ALIS) testing. DOT&E's open recommendations deal with mission data load test integration, the helmet mounted display system, fueldraulics' survivability, VSim verification, aircraft repair times, Block 2B weapons delivery accuracy, and F-35B STOVL fielding concerns. Most of GAO's closed recommendations to the JPO dealt with cost and reschedule baselining, maintaining expected funding, monitoring software, and F-35B progress. GAO's open recommendations deal with executing knowledge-based, evolutionary acquisitions and limiting production strategies. The open recommendations by the Department of Defense Inspector General (DoD IG) to the JPO deal with establishing quality assurance programs over the contractor, which the JPO disagrees with because it does not have the resources or the responsibility to perform this through the supply chain (DoD IG, 2013).

Figure 2 shows three dense reports'/journals' co-occurrence/c-coefficient tables generated by ATLAS.ti QDA software; these were explained in depth by the dissertation this article is based upon. The table on the left shows the associations between acquisition strategy challenges and the pillars of acquisition. The middle table shows the interassociations within acquisition strategy challenges. The table on the right shows the interassociations within pillars of acquisition. For each table, each coded component shows the number of codes in parentheses along the header rows and columns. At each intersection, the co-occurrences and c-coefficients are paired. The intent here is to show some specific examples to eventually discern some obvious general trends. Data that are covered up by overlapping tables are not significant.

The left table in Figure 2 has acquisition strategy challenges' component codes depicted in the left column and the pillars of acquisition component codes depicted in the top row. Strong associations have a c-coefficient [greater than or equal to] 0.08 and with two co-occurrences. Circled in red, out of the 27 co-occurrences between TN (Technology Negative) and SLR (Schedule Lengthened Reason), eight co-occurrences blamed design immaturity, eight co-occurrences blamed the lack of flight-testing assets that lengthened schedules, six co-occurrences blamed complex software, and three co-occurrences blamed weight control. In the middle table, acquisition strategy challenges interassociations, the one example circled in green, three co-occurrences between BRP (Balancing Requirements Positive) and CMP (Commonality Positive) pointed to Joint Services' expected savings from economies of scale. In the right table, pillars of acquisition interassociations, the one example circled in red, out of the 15 co-occurrences between CI (Cost Increase) and SL (Schedule Lengthened), 12 co-occurrences indicate that schedule delays preceded cost growth, but six out of the 12 pointed to technology challenges being the root cause.

Figure 3 shows percent growth or decline in JSF F-35 reported program cost, unit cost, and acquisition cycle time (official program start to full-rate production) between 2001 and 2013 as reported by the General Accounting Office and GAO. The percentage represented in vertical bars is based on fiscal year (FY) 2015 dollars, but 2001 is the base year shown by 0% change. The 2013 callout box is also in FY 2015 dollars. After the large spike in unit cost in 2003, the JSF program was restructured, but the program triggered a Nunn-McCurdy breach in 2007 for exceeding the 2003 restructured baseline for total program cost and again in 2010 for exceeding the 2007 restructured baseline (GAO, 2014). From 2001 to 2013 in FY 2015 dollars, the JSF program's total program cost grew 49.7%, its program unit cost grew 74.7%, and the acquisition cycle time increased 104.3%, doubling the time (General Accounting Office 2003, 2004; Government Accountability Office 2005-2011a, 2012a, 2013-2015).

Questionnaire Responses from JSF SMEs

The breakdown of questionnaire respondents was slightly different than planned because potential participants were coming and going while the author was coordinating for DoD Information Management Control Office's (IMCO) approval for 4 months. Seventeen questionnaires were sent to the managers at the JPO, 25 to the customers from the military services, and eight to the providers, the contractors at LM. If there had been more than 10 providers (LM contractors), that would have incurred more IMCO requirements because it would have constituted a public survey.

Each participant was allowed 4 weeks beginning in late March of 2015. A 50% response rate was expected to have purposeful sampling according to Zikmund, Babin, Carr, and Griffin (2010) and Creswell (2007). Of the 50 potential participants, 42 participants responded in April 2015 for an incredible 86% response rate--10 out of 17 from the JPO, 25 out of 25 from the Services, and seven out of eight from LM (see Table 3). However, it took 5 weeks for the seven LM responses to be technically reviewed and legally released by the JPO Security Office and lawyers.

Table 4 speaks to the credibility of the respondents. From the top of Table 4, although there were only seven LM contractor ("Con") respondents due to not making the questionnaire a public survey, active duty (AD) and general schedule (GS) civilian respondents were practically even at 17 and 18, respectively. A majority (34 of 42 or 81%) of the respondents had earned a master's degree as their highest level of education. Most of the respondents were either executives/directors, program managers (of the variant or subsystem), or action officers with a few dedicated pilots, and a tester. A strong majority (69%) of the respondents considered themselves SMEs, a third of the respondents had pilot experience, a few considered themselves as maintainers (12%), and several (19%) as policy deciders. By far, the providers from LM had the most years of experience in dealing with each acquisition strategy--40% more than the managers at the JPO, and more than double the customers from the Services. Overall, respondents had the most years of experience when dealing with Harnessing Technology, followed by Balancing Requirements, followed by a near three-way tie between the remaining strategy challenges. The large standard deviations in years of experience was because several respondents (12 of 42, 29%) had zero experience with more than one associated acquisition strategy challenge, but they were all highly recommended by their superiors to be invited to participate in the study.

Similar to "years' experience" from the bottom of Table 4, Table 5 shows the average amount of words respondents used to answer the questions. The customers had the most to say at 1,520 words per an entire set of answers, almost 70% more than the managers, and almost twice as much as the providers. This order held true for the number of words per each acquisition strategy question except for commonality, where the managers and providers switched places for second and third.

Respondents had the most to say about Balancing Requirements, with an average of 114 words for that section of three questions, followed by Harnessing Technology and Encouraging Partnering, with 89 and 83 words per section, respectively, followed by Evoking Concurrency and Demanding Commonality, with 75 and 55 words, respectively. The standard deviations were also large, but the quality of answers should not be attributed to the number of words.

Table 6 shows the breakdown of family codes by the type of respondent-manager (JPO), customer (Services), or provider (LM). Out of 1,564 respondent codes, customers from the Services provided the most codes, but they were proportional to being the largest group of respondents. Customer respondents' codes were about 2.5 times more than the managers from the JPO and four times more than the providers from LM. These proportions held throughout the acquisition strategy challenges' codes, indicating consistency in coding. Although the proportional relationship of total coding versus the family codes of cost, schedule, and performance did not stand up as well, the coding proportions within cost, schedule, and performance were consistent. However, for all coding, remaining objective was challenging for the researcher and required an iterative process of recoding to ensure that the respondents were actually commenting about an acquisition strategy challenge in its relation to one or more of the pillars of acquisition. It is interesting to point out that Balancing Requirements and Harnessing Technology had the most negative codes, with 91 and 89. However, Balancing Requirements and Demanding Commonality had the most positive codes, with 62 and 56. It is interesting that all the acquisition strategy challenges have nearly the same number of recommendations coded-about 50. Within the pillars, the positive categories of Cost Decrease and Cost Decrease Reason were coded the most with 79 and 77 codes. On the negative side, Performance Reduced and Cost Increase Reason had the highest codes with 68 and 52.

Similar to Figure 2, Figure 4 shows three questionnaire respondents' co-occurrences/c-coefficient tables generated by ATLAS.ti QDA software; these were explained in depth by the dissertation this article is based upon. The table on the left shows the associations between acquisition strategy challenges and the pillars of acquisition. The middle table shows the interassociations within acquisition strategy challenges. The table on the right shows the interassociations within the pillars of acquisition. For each table, each coded component shows the number of codes in parentheses along the header rows and columns. At each intersection, the co-occurrences and c-coefficients are paired. The intent here is to show some specific examples to eventually discern some obvious general trends. Data that are covered up by overlapping tables are not significant.

The table on the left of Figure 4 has acquisition strategy challenges' component codes depicted in the left column and the pillars of acquisition component codes depicted in the top row. There were several more strong associations circled for this analysis lane as compared to the Reports'/ Journals' lane, and each strong association was explained in depth in the dissertation this article is based upon. For the green circle in the middle-left--14 of the 21 co-occurrences between CMP (Commonality Positive) and CD (Cost Decreased) believed there has been and will continue to be cost savings. For the red circle just below, all 16 co-occurrences between CCN (Concurrency Negative) and CIR (Cost Increase Reason) understood that all the early jets, around a 100, need significant and costly modifications that exceeded expectations. For the orange circle to the right between CCR (Concurrency Recommendation) and CR (Cost Recommendation), the c-coefficient required further investigation and yielded one of the best recommendations from a manager, "Maintain program discipline up front, don't over-promise" in the ability to maintain planned concurrency. For the second green circle at the bottom, 14 of the 18 co-occurrences between PTP (Partnering Positive) and CDR (Cost Decrease Reason) reiterated the savings of having multiple countries investing in development and reducing the price per unit. The remaining four occurrences pointed out that Partner countries were given economic opportunities to produce or maintain significant aspects of the F-35.

In the middle table, acquisition strategy challenges interassociations, for the red circle, 8 of the 12 co-occurrences between BRN (Balancing Requirements Negative) and CMN (Commonality Negative) emphasized the resulting differences and uncompromising tradeoffs between the F-35 variants in terms of range requirements, fuel capacities, and weights. In the right table, pillars of acquisition interassociations, for the green circle, the 19 co-occurrences between CD (Cost Decreased) and SS (Schedule Shortened) read more like revisionist history that has not occurred yet-seven of 19 co-occurrences praised commonality based on future expectations that commonly designed parts will pay off huge dividends in lower sustainment costs for 40 more years of the program and that common mission systems (mission planning and sensors) will reduce the need for Services and other partner nations to look for expensive alternatives. Another seven co-occurrences opined that balancing requirements would have produced earlier savings if no changes in requirements were made. Although three co-occurrences believed that concurrency is currently successful in reducing cost and time, its current reputation is quite the opposite. For the red circle, out of the 31 co-occurrences between CI (Cost Increase) and SL (Schedule Lengthened), 12 were attributed to immature technology, 10 to balancing requirements, three to commonality, four to concurrency, and two to partnering--all five of the acquisition strategy challenges the 2015 study focused on.

Answering the Research Questions

In answering the main research question, the 2015 study helped us understand to a deeper and richer level that acquisition strategies have both aided and hindered joint aircraft programs' schedule, cost, and performance in terms of Balancing Requirements, Harnessing Technology, Demanding Commonality, Evoking Concurrency, and Encouraging Partnering. The following sections will answer each subquestion (SQ) about the acquisition strategy's original intent, its positive and negative effects on schedule, cost, and performance, and what improvements could be made. Individual acquisition strategy challenge taxonomies will show relationships between strategies and pillars, and emphasize relativeness of what was learned from each analysis lane from the 2015 study, and together will help summarize lessons learned.

For each acquisition strategy challenge's summary taxonomy, there will be contributing factors and arrows in the appropriate color matching to the previous synthesized Figure 1 or to the previous co-occurrence/c-coefficient tables in Figures 2 and 4. The previous joint aircraft programs' analysis lane will be depicted in brown, government and think tank reports and scholarly journals will be in purple, and respondents' questionnaires will be in black. The components for acquisition strategy challenges and pillars of acquisition will usually be in red/negative or green/positive boxes if they emanated from a strong association starting with an acquisition strategy challenge. Otherwise, some component boxes will remain clear. Within the tan circle of acquisition strategy challenges, for strong relationships to other strategies, only contributing factors from the analysis lane from which they came from will be shown in the appropriate colored text. Within the olive area, contributing factors between pillars of acquisition components will be shown in white text over the appropriate colored arrow to also indicate from which analysis lane it came.

SQ1 on Balancing Requirements

The original intent of balancing requirements, especially with joint aircraft programs, is to merge and trade off the needs of several U.S. military services and nations into one weapon system, thus combining several programs into one in order to save money and time as opposed to each Service or country developing its own aircraft program. The result of Balancing Requirements would be a weapon system that has commonality in design, structure, mission systems, and parts among possible variants. Specifically, the JSF program initially combined eight U.S. military services' programs: Advanced Short Take-Off/Vertical Landing, the Short Take-Off/Vertical Landing Strike Fighter, the Common Affordable Lightweight Fighter, the Multi-Role Fighter, the Advanced Tactical Aircraft, the Naval Advanced Tactical Fighter, the Advanced Attack/Advanced/Fighter Attack, and the Joint Advanced Strike Technology programs (Antill & Ito, 2013); and then the JSF program became the leading fighter replacement program candidate for many foreign nations.

Balancing capability requirements can have a positive effect on cost by combining several programs into one and by getting years of sustainment savings due to common parts over a very large pool of aircraft. In some respect, Balancing Requirements reduces schedule because of the one single design effort and the consolidation of test points between variants. On the other hand, balancing capability requirements has had a negative effect on the JSF schedule because it has taken longer than expected for the Services to agree on requirements since they operate in different environments. Although commonality is great in the long term for parts replacement, in the short term, the JSF program took more time and money to design those common parts within specified weight criteria, especially to meet the requirements of the USMC's F-35B, which took a SWAT (STOVL Weight Attack Team) to solve. During the time it took for the Services to compromise and the engineers to design, Services updated and added key performance parameters (KPP) as the threat evolved, thereby increasing cost.

To improve the process of balancing capability requirements, one can either force performance-affordability compromises between the Services earlier and more often or be willing to diverge from the designs between the variants earlier. As we learned from questionnaire respondents only, initial airframe commonality expectation was 80%, and this was reduced in actuality to 30 to 40% by 2015. However, respondents highlighted that mission systems and avionics between all variants were between 95 and 100%. The USMC gave up time, but still got their STOVL replacement to the AV-8 Harrier. The USMC did not care as much about low observability as the USAF did, so much so that the USMC ditched the internal gun for an external gun pod that increases the radar cross section. Although the USAF's desire for an internal gun conflicted directly with the USMC's need for a vertical lift fan in almost the same location of the aircraft, the USAF did lose speed and range performance to incorporate other aspects that were needed for USMC STOVL engine capability.

As a matter of relativeness between the analysis lanes, Figure 5 shows a summary taxonomy of all the Balancing Requirements components' contributing factors and how they affect the pillars of acquisition and how the pillars of acquisition react to one another under the guise of balancing requirements. We learned from previous joint aircraft programs (in brown), especially from the 1960s TFX F-111, that combining multiple Services' requirements is a difficult challenge. The difference with the JSF program is that the USMC needs the F-35B more now than the USN needed the F-111 back then. Numerous negative contributing factors, mostly from the previous joint aircraft program analysis (in brown) and from the respondent questionnaire analysis (in black), are more prevalent than positive contributing factors, especially in relation to cost, which RAND concluded cost more for joint programs than for multiple single-Service programs (Lorell et al., 2013). The relationship that stands out within the pillars of acquisition is that overpromised performance, coupled by underestimated cost and schedule, resulted in Services' capability tradeoffs.

SQ2 on Harnessing Technology

The original intent of harnessing technology is for the weapon system to give warfighters an asymmetric advantage over their potential adversaries. Although potential adversaries have developed low-observable (LO) cruise missiles, the United States is considered the leader in aircraft LO development. That development started with the F-117 stealth fighter and continues with the B-2 stealth bomber and the F-22 fighter--which are being slowly countered by potential adversaries. As reported in annual DOT&E reports (2004-2014), the F-35 fighter is expected to be the most advanced and dominant aircraft in the world for decades to come, not just for its next generation LO stealth technology, but for its fused and integrated sensors, weapons, and electronic attack used in conjunction with LO.

Harnessing technology is not cheap, and it takes time, but it eventually improves tactical performance. Even with COTS technology, the military services eventually manipulate the technology to meet military specifications much like what happened to the 1990s JPATS T-6. In conjunction with concurrency, numerous advanced technologies, even if somewhat immature, can be brought to the warfighter quicker by flying with the systems sooner to fix problems earlier so aircrews can have the chance to survive against an advanced threat. What makes the F-35 and the F-22 "5th Gen" is not just its low observability, but its fusion of sensors. The F-35 uses LO stealth structure, active-electronically scanned array (AESA) radar, and electronic warfare that has incrementally improved over the last 20 years. Although air forces have used helmets with displays in the visor for over 15 years, the JSF's helmet mounted display system (HMDS) is so advanced that you cannot employ the F-35 without it, and if it fails, the pilot returns to base. The latest and greatest technologies are the electro-optical targeting system (EOTS; forward looking air-to-air and air-to-ground infrared search and track system) and the distributed aperture system (a 360-degree day/ night electro-optical system), which give the pilot a protective sphere of situational awareness of incoming aircraft and missile threats.

Making sure these technologies work as advertised usually takes more time and money than planned. Much like the 1960s TFX F-111 variable wing sweep (for high and low speeds) and variable inlet (for the new engine) and the 1980s JVX V-22 tilt rotor technology challenges, the JSF F-35 experienced many technology challenges just to fit the unique lift fan engine design for the USMC F-35B STOVL into the same basic airframe mid-section for all F-35 variants (DOT&E, 2010)--part of the negativeness of commonality. Not only was the F-35B overweight by several hundred pounds for quite some time before a solution was found (Blickstein et al., 2011), but the HMDS, which is exactly the same for all variants, has also been too heavy by ounces to be safe during ejection, and had a jittery display that was fixed separately (GAO, 2014). Concerning academics courseware for pilot and maintenance training and for aircraft maintenance tracking records, ALIS is still immature and requires workarounds that use legacy academic courseware and legacy maintenance-tracking systems (Everstine, 2015). The root causes to all these late schedules and cost increases are due to the complex and immature software that is still being developed, written, and tested. Many of the requirements' developers within each Service were overly optimistic about the envisioned capabilities and the time and cost it would take to deliver. As much as concurrency helped this endeavor, more redesigns and rework than expected occurred during initial flight training.

Most of the improvements recommended for the JSF were shown in Table 2 to correct technical deficiencies. Unclosed items include software development, the HMDS, weapons accuracy due to AESA and EOTS issues, fueldraulics' survivability, and fire suppression. All of these were pursued for the USAF's IOC declaration in August 2016, five years later than planned (Insinna, 2016; Pike, 2012). On the positive side, for any great leaps in technology for the United States to maintain tactical advantages in aerial warfare, we just have to try--because even temporary failures produce gains in knowledge.

As a matter of relativeness between the analysis lanes, Figure 6 shows a summary taxonomy of all the Harnessing Technology components' contributing factors and how they affect the pillars of acquisition and how the pillars of acquisition react to one another under the guise of "mature" technology. Figure 6 is visually significant in showing that all analysis lanes (in brown, purple, and black) contributed several factors to Technology Negative (TN) technology that had numerous negative effects on schedule, cost, and performance. As for recommendations, the green/positive and red/ negative Technology Recommendations (TR) point to a split Performance Recommendation (PR). Within the pillars of acquisition olive triangle, although there may be a chance that new technologies can significantly improve performance and then reduce cost and decrease schedule, it is more likely that these new technologies will be overpromised, will underperform, and will result in deficient capabilities due to immaturity with underdeveloped software and knowledge gaps. Government program managers need to do a better job in assessing technology readiness levels of new systems.

SQ3 on Demanding Commonality

The original intent of demanding commonality is to drive down development, manufacturing, and sustainment costs by having a high level of interchangeable parts and software for a weapon system to reap the benefits of economies of scale when large quantities are bought. Commonality was the result of Balancing Requirements among three Services into a single, multivariant aircraft that has about 40% airframe commonality, but almost 100% commonality in mission systems, sustainment, and training software, eventually supporting over 3,000 F-35s. Although the airframe commonality among the F-35A/B/C is not as high as the 80% expected, there is still just one manufacturing line that can produce all three variants, significantly lowering the price per unit, according to annual GAO reports, which became attractive to foreign nations. Furthermore, with only 40% airframe commonality, most of the parts that can be replaced are common and that will save hundreds of millions of dollars over the decades to come. As more F-35s are built, over time there are less spare parts per aircraft needed, which drives logistical costs down. Even with partner-unique parts, the commonality of the computerized maintenance and training system based on the ALIS backbone will also save hundreds of millions of dollars once ALIS has its full capability. Furthermore, beyond the airframe, with almost 100% common mission and training systems, combining tests points shortens the schedule.

Similar to the Balancing Requirements previous discussion in terms of negativeness, in order to meet requirements within the different Services' operating environments, compromise entailed making sacrifices in KPPs. For example, the USAF accepted a less than optimal ejection system with a backwards canopy and less fuel capacity to meet USMC F-35B weight needs. When compromise was not attained, divergence occurred. For example, the USAF F-35A maintained the internal gun, while the USMC F-35B removed the internal gun to make more room for the lift fan part of their engine, and the USN F-35C has a third larger wing than the F-35As and Bs. These separate designs contradict the ideal of demanding commonality. A common recommendation from the managers and the providers is to keep forcing performance-affordability compromises early in the program, but even the customers admitted that Services' autonomy and parochialism usually win at a cost of development time.

As a matter of relativeness between the analysis lanes, Figure 7 shows a summary taxonomy of all the Demanding Commonality components' contributing factors, and how they affect the pillars of acquisition, and how the pillars of acquisition react to one another under the pressure of commonality. Figure 7 is a visual "draw" between the positives and negatives, looking very similar to those of Balancing Requirements in Figure 5. Most of the positive factors and relationships, however, came from the questionnaire responses (in black), and the negatives came from previous joint aircraft programs (in purple). If commonality was graded over time, it has been losing early and often, evident by a couple of Nunn-McCurdy breaches, but GAO reports and several respondents believe there is high probability that over time the story will change to be a successful strategy. Within the pillars of acquisition olive triangle, Demanding Commonality was sold as a cost-and time-savings strategy to make three Service fighter programs into one, while the same mission systems and simplified parts pool will eventually reap a cost benefit. However, the expectations of these savings were oversold in the near term, and the Services will not let you forget that they sacrificed on some KPPs to attempt to meet the commonality goals.

SQ4 on Evoking Concurrency

The original intent of evoking concurrency was to systematically overlap development, production, test, and fielding of a weapon system to get it quickly to the warfighters with some capability in order to improve follow-on production lots based on what was learned from the warfighters, thus saving time and money. Concurrency was offered as a strategy to answer the 1980s Packard Commission about how to prevent long and costly defense acquisition programs (Eide & Allen, 2012). In theory, concurrency shortens the program by having test and training conducted simultaneously. It should also force the program to account for supportability and logistics earlier. Several respondents explained how concurrency was planned for 4 to 6 years over three blocks of aircraft--Blocks 1, 2, and 3, but the program ended up with 11 different blocks--Blocks 0, 1A, 1B, 2A, 2B, 3i, 3F, 4.1, 4.2, 4.3, and 4.4. By 2011, the JPO's plan for U.S. F-35s was to retrofit 25% of the fleet based on procuring 600 aircraft before the end of initial operational test and evaluation in March 2012 (Blickstein et al., 2011). In comparison, the F-111 produced 141 F-111s for the USAF before changes could be made on the manufacturing line for a 25% concurrency rate against 547 total USAF F-111s (Coulam, 1977; Richey, 2005). The final concurrency rate will not be known until after the fact.

The 1980s JVX V-22 relied heavily on concurrency. It entered full-rate production without ensuring mature manufacturing processes that required a redesign and retrofit of hydraulics and electrics for the tilt-rotor system, resulting in a Nunn-McCurdy breach in 2001, after the USMC had begun MV-22 aircrew training in 2000 (General Accounting Office, 2003; Whittle, 2010). Seven years later, the USMC declared MV-22 IOC in 2007 (Whittle, 2010). There is little argument that with concurrency more F-35 issues have been found earlier than otherwise predicted, and many issues have been corrected quicker with cut-ins on the production line, but the amount and complexity of the retrofits exceeded expectations. Most of this disappointment is due to incomplete, late flight testing and overpromised capabilities. Although the USAF started training in 2012 with an aircraft of very limited capability, right after the second and last Nunn-McCurdy breach, IOC will not be declared until 2016. Instead of dealing with three blocks of aircraft during these 4 years, one "operator" respondent emphasized that pilots and maintainers will have dealt with seven blocks--Block 0.5, 1A, 1B, 2A, 2B, 3i (small "i" for initial combat capability), and 3F. Follow-on F-35 development will include four more blocks starting in 2019 with one every 2 years--Block 4.1, 4.2, 4.3, and 4.4 (Tirpak, 2015).

Unfortunately, Concurrency Recommendations (CCR) did not have any strong associations for the Reports'/Journals' co-occurrence/c-coefficient tables in Figure 2, but the left table for the Respondents' co-occurrence/c-coefficient tables in Figure 4 showed they required further investigation. It suggested that program managers should be more disciplined when accepting immature technology and be more suspicious of the promises of capabilities early on. By digging deeper into the respondents' questionnaires, some appropriate recommendations were found. A few manager respondents suggested ensuring the technology readiness levels are more accurate, reserving concurrency for a less complex weapon system, and to never accept this level of concurrency again. Several customer respondents suggested making the early production lots smaller in quantity. One provider said this about JSF concurrency: "The original model for JSF is solid. Build test aircraft, build training aircraft while testing is going on, and then accelerate to full rate production when the aircraft has matured sufficiently."

As a matter of relativeness between the analysis lanes, Figure 8 shows a summary taxonomy of all the Evoking Concurrency components' contributing factors, and how they affect the pillars of acquisition, and how the pillars of acquisition react to one another under the pressure of commonality. Factors contributing to Concurrency Negative (CCN) are more apparent than to Concurrency Positive (CCP). All analysis lanes (previous in brown, reports/journals in purple, and questionnaires in black) significantly contributed to the concurrency negative impacts on schedule, cost, and performance. Within the pillars of acquisition olive triangle, there are dueling positive and negative component triangles between schedule, cost, and performance. The positive Schedule Shortened (SS)-Cost Decreased (CD)-Performance Improved (PI) triangle assumes mature technology and expects to wring out problems sooner to get a better aircraft sooner. The negative Schedule Lengthened (SL)-Cost Increased (CI)-Performance Reduced (PRD) triangle harps on overpromised results and underperforming reality that was reiterated by the reports/journal (in purple) and the respondents' questionnaire analyses (in black).

SQ5 on Encouraging International Partnering

The original intent of encouraging partnering was to get partner nations invested early in the program to help pay for development and to order and receive aircraft early in the program. This would reduce the price per unit and result in coalition interoperability. In return for taking risk by investing in development, the Partners get to learn how to develop a 5th Gen aircraft that no other opportunity would provide. Over the life of the 40-year program, sustainment costs are also shared. As a business model to defend against U.S. congressional budget cutters, international support goes a long way, especially when the expectation is to train and fight together.

Most respondents referred to building partner capacity as the positive by-product of partnering--it promotes the United States' interest in not fighting alone, in fighting on the same level on the same side, and in spreading the burden of policing the world. Besides reducing U.S. development and production costs and building partnership capacity in the expectation we will fight together, partner nations reap the added benefits of economic opportunity in their own countries as their companies become suppliers for LM. As one of the first partners, U.K.'s Martin-Baker will manufacture all the F-35A/B/Cs' ejection system, representing billions of dollars in revenue to outfit over 3,000 potential aircraft (JPO, 2014). Italy gained 10,000 jobs created by their industrial participation in the program including the activities of the Cameri Final Assembly Check-Out, Maintenance, Repair, Overhaul, and Upgrade (MRO&U) plant to assemble their 90 F-35s via license from LM (Marrone, 2013). Even Turkey was approved to be the first of three nations to oversee MRO&U for the F135 engine (Butler, 2015). Partner countries also get to bid on managing spare parts logistics hubs for Europe and Asia (Butler, 2015).

This arrangement, guided by the JPO (2007) Production, Sustainment, and Follow-on Development (PSFD) Memorandum of Understanding (MOU) signed by the partner nations listed in the MOU's Table 3 (Chapter 2), is a first-of-a-kind partnership that comes with several challenges not experienced in normal FMS. In FMS cases, the weapon system is already developed, there is extra capacity to produce and to train if requested, and the United States usually dictates the terms. In this partnership, no matter how many aircraft a nation is purchasing, the foreign countries get an equal vote right from the start, and when priorities cannot be agreed upon, it takes longer. If a country has a unique requirement, although it pays for it alone, it takes nonrecurring engineering time away from other design and test priorities. But the most difficult challenge concerning the strong relationship between cost and schedule is the problem of getting classified capabilities released to partner nations. This is especially important when USAF and foreign F-35A student pilots will be training with one another, taught by one another by USAF and Partner instructors, and flying each other's aircraft at Luke AFB--a process known as "pooling." Although there are DoD offices in charge of protecting critical technologies, five customer respondents commented that other disclosure, security, and exportability offices have not fully embraced or understood the PSFD MOU's intent. Improvements to remedy these negatives come strictly from respondents' questionnaires and start with recommending delaying partnering until development is complete, settling disclosure issues before partners invest, and making voting proportional to the number of aircraft ordered.

As a matter of relativeness between the analysis lanes, Figure 9 shows a summary taxonomy of all the Encouraging Partnering components' contributing factors and how they affect the pillars of acquisition and how the pillars of acquisition react to one another under the pressure of commonality. Although Figure 9 looks visually even, the contributing factors to Partnering Positive (PTP) of building partner capacity, interoperability, and true coalition warfare, and the tangible, international economic opportunities brought forward mostly by the reports/journals (in purple) and the respondents' questionnaires (in black), outweigh the procedural frustrations in Partnering Negative (PTN) of releasability and voting that will be solved over time by necessity. However, there will be a price to pay in time and cost within the pillars of acquisition olive triangle. This is especially true with unique country requests and the fact that partner nations all have civilian governments that may delay ordering and purchasing from the previously agreed plan, which instantly raises the price per unit that particular year.

Final Recommendations

All program managers in the defense industry and in commercial business should note the relations within the pillars of acquisition that were confirmed. Although the JSF program is in the 21st century, lessons on the pillars of acquisition were first relearned in the 20 th century with the TFX, JVS, and JPATS programs and reconfirmed by analysis on the JSF program. If technical performance cannot be sacrificed, but is hard to achieve, expect cost to rise and schedule to lengthen. If cost is fixed due to budgeting and the schedule is expected to stay the same, performance will suffer and requirements will be lowered. If a program manager wants to maintain schedule, cost usually goes up for more personnel, but performance may suffer again due to rushing.

What is learned is not necessarily about scandal and tragedy; however, all program managers in the defense industry will recognize familiar lessons and learn some new ones:

* Balancing requirements between the Services is not recommended for large joint MDAPs, especially for aircraft. However, balancing requirements within a single-Service program to meet cost, schedule, and performance goals should be expected. This takes disciplined leadership and the ability to manage expectations with transparency of what can be achieved.

* Harnessing technology, or reaching for immature technology, should be done only if needed to maintain a tactical advantage in a strategic environment. Otherwise, program managers need to assess technology readiness levels better and not be so enamored with contractors' glossy brochures of cost, scheduling, and performance.

* Demanding commonality may be costly and time consuming as the result of balancing requirements in the near term, but should pay off with a streamlined logistics system in the long term. Do not underestimate the commonality of mission systems (communications, sensors, mission planning, and prognostic health management) and of training systems (courseware and simulators).

* Evoking concurrency as a primary strategy will be disappointing if technology was incorrectly represented as mature, but this is a matter for better discipline, management, and execution, and it cannot be addressed with Flyvbjerg's (2003) megaprojects' paradox that the program is too big to fail. If concurrency occurs because a development program falls behind schedule, project managers need to adjust expectations, but that still requires discipline.

* Encouraging partnering needs to be understood better from the beginning and embraced by government agencies and military services in terms of disclosure, security, and exportability-especially if the United States truly wants to build partner capacity and to have true, interoperable, coalition warfare.

Col Roger Witek, USAF (Ret.)

References

Air Education and Training Command. (2010, March). A training system for the 21st century: JPATS and the T-6A (AFD-061213-005). Retrieved from https://www. rairarubiabooks.com/related-pdf-enter-trainer-extended-wings.html

Antill, P., & Ito, P. (2013). The UK and the Joint Strike Fighter: The trials and tribulations of international collaborative procurement. International Journal: Canada's Journal of Global Policy Analysis, 68(1), 13-29. doi:10.1177/002070201306800103

Art, R. (1969). The TFX decision: McNamara and the military. Boston, MA: Little, Brown and Company.

Associated Press. (2016, April 26). Senator says fighter program has been scandal and tragedy. Retrieved from http://www.foxnews.com/us/2016/04/26/senator-saysfighter-program-has-been-scandal-and-tragedy.html

Blickstein, I., Boito, M., Drezner, J., Dryden, J., Horn, K., Kallimani, J., ... Wong, C. (2011). Root cause analyses of Nunn-McCurdy breaches: Zumwalt Class Destroyer, Joint Strike Fighter, Longbow Apache, and Wideband Global Satellite (Vol. 1). Retrieved from RAND Corporation website: http://www.rand.org/content/dam/rand/pubs/ monographs/2011/RAND_MG1171.1.pdf

Blickstein, I., Nemfakos, C., & Sollinger J. M. (2013). Digging out the root cause: Nunn-McCurdy breaches in major defense acquisition programs. Defense Acquisition Research Journal, 20(2), 128-153. Retrieved from http://www.dau.mil/ publications/DefenseARJ/ARJ/ARJ66/ARJ_66-Blickstein.pdf

Butler, A. (2015, January 6). F-35 nearing in-service debuts: With a handful of fighters nearing operations, F-35 is still the one to beat. Aviation Week and Space Technology, 176(47), 69. Retrieved from http://aviationweek.com/defense/j-10bf-35-nearing-service-debuts

Coulam, R. (1977). Illusions of choice: The F-111 and the problem of weapons acquisition reform. Princeton, NJ: Princeton University Press.

Creswell, J. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks, CA: Sage.

Defense Acquisition University. (2006). Risk management guide for DoD acquisition (6th ed.). Retrieved from http://www.dau.mil/publications/publicationsDocs/ RMG%206Ed%20Aug06.pdf

Department of Defense. (2012). Joint Primary Aircraft Training System (JPATS) Selected Acquisition Report (SAR) (RCS: DD-A&T(Q&A)823-560). Retrieved from http://www.globalsecurity.org/military/library/budget/fy2012/sar/jpats_ december_2012_sar.pdf

Department of Defense Inspector General. (2013, September 30). Quality assurance assessment of the F-35 Lightning II program. Retrieved from http://www.dodig. mil/pubs/documents/DODIG-2013-140.pdf

Director, Operational Test and Evaluation. (2004). Fiscal year2004 annual report F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/ reports/FY2004/pdf/af/2004F-35JSF.pdf

Director, Operational Test and Evaluation. (2005). Fiscal year 2005 annual report F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/ reports/FY2005/pdf/af/2005f-35.pdf

Director, Operational Test and Evaluation. (2006). Fiscal year 2006 annual report F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/ reports/FY2006/pdf/af/2006f35.pdf

Director, Operational Test and Evaluation. (2007a). Fiscal year 2007 annual report F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/ reports/FY2007/pdf/af/2007f35.pdf

Director, Operational Test and Evaluation. (2007b). Joint Primary Aircraft Training System (JPATS). Retrieved from http://www.dote.osd.mil/pub/reports/FY2007/ pdf/navy/2007jpats.pdf

Director, Operational Test and Evaluation. (2008). F-35 Lightning II Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/reports/FY2008/ pdf/dod/2008f35.pdf

Director, Operational Test and Evaluation. (2009). F-35 Lightning II Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/reports/FY2009/ pdf/dod/2009f35.pdf

Director, Operational Test and Evaluation. (2010). F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/reports/FY2010/pdf/ dod/2010f35jsf.pdf

Director, Operational Test and Evaluation. (2011). F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/reports/FY2011/pdf/ dod/2011f35jsf.pdf

Director, Operational Test and Evaluation. (2012). F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/reports/FY2012/pdf/ dod/2012f35jsf.pdf

Director, Operational Test and Evaluation. (2013). F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/reports/FY2013/pdf/ dod/2013f35jsf.pdf

Director, Operational Test and Evaluation. (2014). F-35 Joint Strike Fighter (JSF). Retrieved from http://www.dote.osd.mil/pub/reports/FY2014/pdf/ dod/2014f35jsf.pdf

Dunne, T. (2011). A case for the F-35 Lightning. National Defence and the Canadian Armed Forces, 11(4), 55-60. Retrieved from http://www.journal.forces.gc.ca/vo11/ no4/55-dunne-eng.asp

Eide, P. K., & Allen, C. D. (2012). The more things change, acquisition reform remains the same. Defense Acquisition Research Journal, 19(1), 99-120. Retrieved from http://www.dau.mil/pubscats/PubsCats/AR%20Journal/arj61/Eide61.pdf

Ergas, H. (2009). Trillions for military technology: How the Pentagon innovates and why it costs so much. Agenda: A Journal of Policy Analysis and Reform, 16(2), 9598. Retrieved from http://search.proquest.com/docview/209796602?accountid= 28180

Everstine, B. (2015, April 25). Problems plaguing F-35's next-gen maintenance system. Retrieved from http://www.airforcetimes.com/story/military/2015/04/15/ problems-facing-f35-maintainers-automated-system/25781075/

Flyvbjerg, B. (2003). Megaprojects and risk: An anatomy of ambition. Cambridge, UK: Cambridge University Press.

Friese, S. (2013). ATLAS.ti 7 user guide and reference. Berlin, Germany: ATLAS.ti Scientific Software Development. Retrieved from http://atlasti.com/wp-content/ uploads/2014/05/atlasti_v7_manual_201312.pdf?q=/uploads/media/atlasti_v7_ manual_201312.pdf

Gantt, M. E. (2002). Strapping in and bailing out: Navy and Air Force joint acquisition of aircraft. Retrieved from Defense Technical Information Center website: http:// www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA420511

Gates, R. (2009, December 2). Statement on Afghanistan to the Senate Armed Services Committee. Retrieved from http://archive.defense.gov/speeches/ default.aspx?mo=12&yr=2009

General Accounting Office. (1973). F-111 Aircraft. Retrieved from http://www.gao.gov/ assets/80/78812.pdf

General Accounting Office. (1986). DoD acquisition: Case study of the Navy V-22 Osprey joint vertical lift aircraft program (GAO-NSIAD-86-45S-7). Retrieved from http://www.gao.gov/assets/210/208624.pdf

General Accounting Office. (1990). Navy aviation: The V-22--Progress and problems (GAO/NSIAD-91-45). Retrieved from http://www.gao.gov/assets/220/213295.pdf

General Accounting Office. (1994). V-22 development--Schedule extended, performance reduced, and costs increased (GAO/NSIAD-94-14). Retrieved from http://www.gao.gov/assets/220/218986.pdf

General Accounting Office. (1997). Navy aviation: V-22 cost and capability to meet requirements are yet to be determined (GAO/NSIAD-98-13). Retrieved from http://www.gao.gov/assets/230/224788.pdf

General Accounting Office. (2003). Defense acquisitions: Assessments of major weapon programs (GAO-03-476). Retrieved from http://www.gao.gov/ assets/240/238188.pdf

General Accounting Office. (2004). Defense acquisitions: Assessments of major weapon programs (GAO-04-248). Retrieved from http://www.gao.gov/ assets/250/241798.pdf

Gertler, J. (2011). V-22 Osprey tilt-rotor aircraft: Background and issues for Congress (RL31384). Retrieved from http://www.fas.org/sgp/crs/weapons/RL31384.pdf

Government Accountability Office. (2005). Defense acquisitions: Assessments of selected major weapon programs (GAO-05-301). Retrieved from http://www.gao. gov/assets/250/245884.pdf

Government Accountability Office. (2006). Defense acquisitions: Assessments of selected major weapon programs (GAO-06-391). Retrieved from http://www.gao. gov/assets/250/249468.pdf

Government Accountability Office. (2007). Defense acquisitions: Assessments of selected weapon programs (GAO-07-406SP). Retrieved from http://www.gao. gov/assets/260/258416.pdf

Government Accountability Office. (2008). Defense acquisitions: Assessments of selected weapon programs (GAO-08-467SP). Retrieved from http://www.gao. gov/assets/280/274156.pdf

Government Accountability Office. (2009). Defense acquisitions: Assessments needed to address V-22 aircraft operational and cost concerns to define future investments (GAO-09-482). Retrieved from http://www.gao.gov/ assets/290/289395.pdf

Government Accountability Office. (2010). Defense acquisitions: Assessments of selected weapon programs (GAO-10-388SP). Retrieved from http://www.gao. gov/assets/310/302379.pdf

Government Accountability Office. (2011a). Defense acquisitions: Assessments of selected weapon programs (GAO-11-233SP). Retrieved from http://www.gao.gov/ assets/320/317081.pdf

Government Accountability Office. (2011b). Joint Strike Fighter: Restructuring places program on firmer footing, but progress is still lagging (GAO-11-677T). Retrieved from http://www.gao.gov/assets/130/126245.pdf

Government Accountability Office. (2012a). Defense acquisitions: Assessments of selected weapon programs (GAO-12-400SP). Retrieved from http://www.gao. gov/assets/590/589695.pdf

Government Accountability Office. (2012b). Joint Strike Fighter: Restructuring added resources and reduced risk, but concurrency is still a major concern (GAO-12525T). Retrieved from http://www.gao.gov/assets/590/589454.pdf

Government Accountability Office. (2012c). Joint Strike Fighter: Restructuring added resources and reduced risk, but concurrency is still a major concern (GAO-12525T). Retrieved from http://www.gao.gov/assets/590/589454.pdf

Government Accountability Office. (2013). Defense acquisitions: Assessments of selected weapon programs (GAO-13-294SP). Retrieved from http://www.gao. gov/assets/660/653379.pdf

Government Accountability Office. (2014). Defense acquisitions: Assessments of selected weapon programs (GAO-14-340SP). Retrieved from http://www.gao. gov/assets/670/662184.pdf

Government Accountability Office. (2015). Defense acquisitions: Assessments of selected weapon programs (GAO-15-342SP). Retrieved from http://www.gao. gov/assets/670/668986.pdf

Hunter, A. (1998). TSR2 with hindsight. Brighton, U.K.: RAF Historical Society.

Insinna, V. (2016, August 2). Air Force declares F-35A ready for combat. Defense News. Retrieved from http://www.defensenews.com/story/breakingnews/2016/08/02/f35-ioc-air-force-operational-acc-combat/87948142/

Jogulu, U., & Pansiri, J. (2011). Mixed methods: A research design for management doctoral dissertations. Management Research Review 34(6), 687-701. doi:10.1108/ 01409171111136211

Joint Program Office. (2007). JSF production, sustainment, and follow-on development memorandum of understanding. Retrieved from http://www.jsf.mil/ downloads/documents/JSF_PSFD_MOU_-_07_Feb_07.pdf

Logan, D. (1998). General Dynamics F-111 Aardvark. Atglen, PA: Schiffer.

Leedy, P., & Ormrod, J. (2009). Practical research: Planning and design (9th ed.). Upper Saddle River, NJ: Pearson.

Lorell, M. A., Kennedy, L., Leonard, R. S., Munson, K., Abramzon, S., An, D. L., & Guffey, R. A. (2013). Do joint fighter programs save money? Retrieved from Rand Corporation website: http://www.rand.org/content/dam/rand/pubs/ monographs/MG1200/MG1225/RAND_MG1225.pdf

Marrone, A. (2013). Italy and the F-35: Rationales and costs. International Journal: Canada's Journal of Global Policy Analysis, 68(1), 31-48. doi:10.1177/ 002070201306800104

McCullough, A. (2015, July 20). First ospreys for Japan. Air Force Magazine: Online Journal of the Air Force Association. Retrieved from http://www.airforcemag. com/DRArchive/Pages/2015/July%202015/July%2015%202015/First-Ospreysfor-Japan.aspx

Melese, F., Franck, R., Angelis, D., & Dillard, J. (2007). Applying insights from transaction cost economics to improve cost estimates for public sector purchases: The case of U.S. military acquisition. International Public Management Journal, 10(4), 357-385. doi:10.1080/10967490701683511

Miller, J. (1982). General Dynamics F-111 "Aardvark." Fallbrook, CA: Aero Publishers.

Moustakas, C. (1990). Heuristic research: Design, methodology, and applications. Newbury Park, CA: Sage.

Patton, M. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.

Pike, J. (2011). F-35 Joint Strike Fighter (JSF) commonality. Retrieved from the GlobalSecurity.Org website: http://www.globalsecurity.org/military/systems/ aircraft/f-35-commonality.htm

Pike, J. (2012). F-35 Joint Strike Fighter (JSF) Lightning II program. Retrieved from the GlobalSecurity. Org website: http://www.globalsecurity.org/military/systems/ aircraft/f-35-program.htm

President's Blue Ribbon Commission on Defense Management. (1986). A quest for excellence--final report to the President. Retrieved from http://usacac. army.mil/cac2/CSI/docs/Gorman/06_Retired/01_Retired_1985_90/07_86_ PackardCommission_FinalReport/01_PackardCommission_FinalReport.pdf

Richey, G. (2005). F-111 systems engineering case study. Wright Patterson AFB, OH: Air Force Institute of Technology. Retrieved from https://acc.dau.mil/ CommunityBrowser.aspx?id=37599&lang=en-US

Shank, G. (2006). Qualitative research: A personal skills approach. Upper Saddle River, NJ: Pearson Merrill Prentice Hall.

Summers, R. (1965). Cost estimates as predictors of actual weapons costs: A study of major hardware articles (RM-3061-PR). Retrieved from RAND Corporation website: http://www.rand.org/pubs/research_memoranda/RM3061.html

TFX contract investigation, parts 1-10: Hearings before the Permanent Subcommittee on Investigations of the Committee on Government Operations, Senate, 88th Cong. 1 (1963). Washington, DC: U.S. Government Printing Office.

TFX contract investigation, parts 1-3: Hearings before the Permanent Subcommittee on Investigations of the Committee on Government Operations, Senate, 91st Cong. 2 (1970). Washington, DC: U.S. Government Printing Office.

Tirpak, J. A. (2015, June 1) F-35 block 4 taking shape. Air Force Magazine: Online Journal of the Air Force Association. Retrieved from http://www.airforcemag. com/DRArchive/Pages/2015/June%202015/June%2001%202015/F-35-Block-4Taking-Shape.aspx

Uda, R. T. (2012). Exploring factors contributing to improvised explosive device (IED) terror campaigns in the United States homeland (Publication No. 3530099). Available from Dissertations & Theses @ Northcentral University; ProQuest Dissertations & Theses Global (1085492122). Retrieved from http://search. proquest.com.proxy1.ncu.edu/docview/1085492122?accountid=28180

Wacker, J. (1998). A definition of theory: Research guidelines for different theory-building research methods in operations management. Journal of Operations Management, 16(4), 361-385. doi:10.1016/S0272-6963(98)00019-9

Wicht, A. C., & Crawley, E. F. (2012). Relieving joint pain: Planning government acquisition of complex common systems. Defense Acquisition Research Journal, 19(3), 221-248. Retrieved from http://www.dau.mil/pubscats/PubsCats/AR%20 Journal/arj64/Wicht_Crawley_ARJ63.pdf

Whittle, R. (2010). The dream machine: The untold history of the notorious V-22 Osprey. New York, NY: Simon & Schuster.

Yin R. (1994). Case study research: Designs and methods (2nd ed.). Newbury Park, CA: Sage.

Young, H., Vorwig, P., Lieberman, M., Wigginton, W., House, M., Black, W., & Maravilla, C. (2010). Aerospace and defense industries. The International Lawyer, 44(1), 337-353. Retrieved from http://search.proquest.com/docview/ 645692834?accountid=28180

Zikmund, W., Babin, B., Carr, J., & Griffin, M. (2010). Business research methods. (8th ed.). Mason, OH: Thomas/South-Western.

Biography

Col Roger Witek, USAF (Ret.), is currently the Director of Training and Services for Tapestry Solutions. Dr. Witek was chief of Air Education and Training Command's Joint Strike Fighter Coordination and Training Office, responsible for the pilot and maintenance training programs for the USAF and seven Partner nations. He has a doctorate in business administration, is a former vice-wing and squadron commander, and is a graduate of the USAF Weapons Instructor Course and the School of Advanced Air and Space Studies. As a Secretary of Defense Corporate Fellow, he produced a research report, "Hard Skills, Soft Skills, Savviness, and Discipline-Recommendations for Successful Acquisition" (available at http:// dcmo.defense.gov/Portals/47/Documents/ Products_Services/SecDef_Fellows/08_09__ Boeing_Report.doc).

(E-mail address: rcwitek1989@gmail.com)

Caption: FIGURE 2. REPORTS'/JOURNALS' CO-OCCURRENCE/C-COEFFICIENT TABLES

Caption: FIGURE 4. RESPONDENTS' CO-OCCURRENCE/C-COEFFICIENT TABLES

Caption: FIGURE 5. BALANCING REQUIREMENTS SUMMARY TAXONOMY

Caption: FIGURE 6. HARNESSING TECHNOLOGY SUMMARY TAXONOMY

Caption: FIGURE 7. DEMANDING COMMONALITY SUMMARY TAXONOMY

Caption: FIGURE 8. EVOKING CONCURRENCY SUMMARY TAXONOMY

Caption: FIGURE 9. ENCOURAGING PARTNERING SUMMARY TAXONOMY
TABLE 1. REPORTS'/JOURNALS' FAMILY CODE BREAKDOWN

                                        DOT&E     GAO         GAO
                                        (11)    SAR (13)   F-35 (20)

Balancing Requirements Intent (BRI)       5        0           1
Balancing Requirements Negative (BRN)    12        4           0
Balancing Requirements Positive (BRP)     8        1           0
Balancing Requirements                    6        0           2
Recommendation (BRR)
Balancing Requirements Sub-total         30        5           3
Technology Intent (TI)                    8        2           0
Technology Negative (TN)                 39        38         19
Technology Positive (TP)                 16        15          3
Technology Recommendation (TR)           61        1           8
Harnessing Technology Sub-total          112       55         30
Commonality Intent (CMI)                  0        2           0
Commonality Negative (CMN)                3        0           1
Commonality Positive (CMP)                1        1           0
Commonality Recommendation (CMR)          1        0           1
Demanding Commonality Sub-total           5        3           2
Concurrency Intent (CCI)                  0        3           0
Concurrency Negative (CCN)               13        17         15
Concurrency Positive (CCP)                0        3           1
Concurrency Recommendation (CCR)         11        2           6
Evoking Concurrency Sub-total            24        23         22
Partnering Intent (PTI)                   0        0           0
Partnering Negative (PTN)                 0        1           0
Partnering Positive (PTP)                 0        0           0
Partnering Recommendation (PTR)           2        1           0
Encouraging Partnering Sub-total          2        1           0
Cost Increase (CI)--bad                   0        8          10
Cost Increase Reason (CIR)--why           4        16          8
Cost Decrease (CD)--good                  0        2           1
Cost Decrease Reason (CDR)--why           0        3           3
Cost Recommendation (CR)                  9        3          15
Cost Sub-total                           12        30         36
Schedule Lengthened (SL)--bad             7        12          9
Schedule Lengthened Reason (SLR)--why    13        20          8
Schedule Shortened (SS)--good             0        1           1
Schedule Shortened Reason (SSR)--why      3        3           2
Schedule Recommendation (SR)             21        3           5
Schedule Sub-total                       42        36         25
Performance Reduced (PRD)--bad           16        4           3
Performance Reduced Reason (PRR)--why    10        7           0
Performance Improved (PI)--good           8        8           8
Performance Increased Reason              2        3           2
  (PIR)--why
Performance Recommendation (PR)          20        1           3
Performance Sub-total                    52        21         17
Grand Totals                             279      174         135

                                          Think     Journals   Totals
                                        Tanks (8)     (16)      (68)

Balancing Requirements Intent (BRI)         4          0         10
Balancing Requirements Negative (BRN)       7          2         25
Balancing Requirements Positive (BRP)       0          1         10
Balancing Requirements                      4          1         13
Recommendation (BRR)
Balancing Requirements Sub-total           14          4         56
Technology Intent (TI)                      3          1         14
Technology Negative (TN)                   10          8        114
Technology Positive (TP)                    0          2         36
Technology Recommendation (TR)              7          5         82
Harnessing Technology Sub-total            20          15       232
Commonality Intent (CMI)                    5          4         11
Commonality Negative (CMN)                 11          7         22
Commonality Positive (CMP)                  0          7         9
Commonality Recommendation (CMR)            3          5         10
Demanding Commonality Sub-total            17          21        48
Concurrency Intent (CCI)                    2          1         6
Concurrency Negative (CCN)                  2          3         50
Concurrency Positive (CCP)                  0          0         4
Concurrency Recommendation (CCR)            0          4         23
Evoking Concurrency Sub-total               4          8         81
Partnering Intent (PTI)                     3          6         9
Partnering Negative (PTN)                   4          8         13
Partnering Positive (PTP)                   1          9         10
Partnering Recommendation (PTR)             2          0         5
Encouraging Partnering Sub-total            8          19        30
Cost Increase (CI)--bad                     6          5         29
Cost Increase Reason (CIR)--why             8          10        46
Cost Decrease (CD)--good                    0          1         4
Cost Decrease Reason (CDR)--why             2          5         13
Cost Recommendation (CR)                   13          7         37
Cost Sub-total                             19          28       126
Schedule Lengthened (SL)--bad               2          2         32
Schedule Lengthened Reason (SLR)--why       2          2         45
Schedule Shortened (SS)--good               0          0         2
Schedule Shortened Reason (SSR)--why        0          0         8
Schedule Recommendation (SR)                1          2         32
Schedule Sub-total                          5          6        115
Performance Reduced (PRD)--bad              4          2         29
Performance Reduced Reason (PRR)--why       1          4         22
Performance Improved (PI)--good             0          1         26
Performance Increased Reason                0          2         9
  (PIR)--why
Performance Recommendation (PR)             5          3         32
Performance Sub-total                      10          12       112
Grand Totals                               97         113       798

Note. (X) = number of documents, DOT&E = Director of Operational Test
and Evaluation, GAO = Government Accountability  Office,
SAR = Selected Acquisition Reports

TABLE 2. TRACKING RECOMMENDATIONS TO THE JPO

Recommendations                            2005   2006   2007   2008

DOT&E 05-Have data collection and           O      C
mission replay to evaluate mission
effectiveness

DOT&E 05-Test and Evaluation Master Plan    O      W      W      W
(TEMP) should identify shortfalls in
opposing force/threats

DOT&E 05-Align requirements for each        O      C
block aircraft

DOT&E 05-Develop predictive model on        O      W      W      C
engine performance after "quick dump"
fuel ingestion

DOT&E 05-Reduce fuel ingestion              O      C
vulnerability by improving fuel bladders
around inlet ducts

DOT&E 06-Update issues from recent                 O      C
operational assessment

DOT&E 06-Consider doing Initial                    O      C
Operational Test and Evaluation (IOT&E)
earlier with operationally
representative aircraft

DOT&E 06-Follow May 06 Defense                     O      C
Acquisition Board ideas for Partner
testing

DOT&E 06-Fund adequate full-scale aerial           O      C
target to confirm operational
effectiveness

DOT&E 06-Conduct full-up, live system-             O      W      C
level live-fire ballistic tests on F-35
to determine vulnerability

DOT&E 07-Retain last two system                           O      C
development and demonstration (SDD)
aircraft

DOT&E 07-Ensure labs are resourced to                     O      W
execute verification strategy and to
surge

DOT&E 07-Develop metrics for                              O      W
verification strategy

DOT&E 07-Develop entrance criteria for                    O      W
IOT&E

DOT&E 07-Reinstate dry bay engine fire                    O      C
suppression

DOT&E 07-Reinstate engine fuel ingestion                  O      C
suppression liner

DOT&E 07-Add-Improve cockpit warning                      O      W
lights to F-35B for ballistic damage
before vertical landing

DOT&E 07-Retain engine fueldraulics and                   O      C
liquid cooling shutoff valves

DOT&E 08-Add resources to flight testing                         O
in FY09-11

DOT&E 08-Explain all test changes to                             O
DOT&E

DOT&E 08-Initiate a Test Plan Working                            O
Group

DOT&E 08-Stabilize production and                                O
deliveries of systems needed for OT&E
and training

DOT&E 08-Complete third iteration of the                         O
TEMP

DOT&E 08-Improve verifications simulator                         O
(VSim) to meet adequate verification
testing

DOT&E 08-Restore capability to minimize                          O
fueldraulics spillage from threat-
induced damage

DOT&E 09-Focus delivery efforts on SDD
aircraft

DOT&E 09-By an Operational Test Review
Team, review IOT&E test plan for Block-
3 aircraft systems

DOT&E 09-Have more transparent contract
negotiations for Block-3 test aircraft

DOT&E 09-Verify, validate, and accredit
test labs

DOT&E 10-Assure new flight test schedule
is realistic

DOT&E 10-Evaluate flight test schedule,
executed versus planned

DOT&E 10-Determine impact of technical
issues of helmet-mounted display

DOT&E 10-Assure software integration is
in flight test

DOT&E 10-Verify/validate msn data loads
(MDLs)

DOT&E 10-Redesign On-board Inert Gas
Generating System (OBIGGS) to maintain
oxygen levels below where fire can be
sustained

DOT&E 11-Use event-driven criteria to
begin flt ops

DOT&E 11-Test transonic buffeting

DOT&E 11-Determine impacts of late
structural durability testing

DOT&E 11-Improve spare/resupply for
flight test

DOT&E 11-Survey test plans for
certifications by outside government
agencies

DOT&E 12-Make corrections to Version 4
of TEMP

DOT&E 12-Conduct testing on Autonomic
Logistics Information System (ALIS)

DOT&E 12-Make operational test (OT)
aircraft fully production-
representative

DOT&E 12-Ensure contractor meeting VSim
requirements

DOT&E 12-Assure integrated VSim and MDL
testing

DOT&E 12-Continue PAO shutoff valve
redesign

DOT&E 12-Consider removing fueldraulic
fuses

DOT&E 12-Consider keeping dry bay fire
extinguisher for the Integrated Power
Pack only

DOT&E 12-Determine ballistic event
survival time

DOT&E 13-Account for historical growth
of flight test

DOT&E 13-Get VSim data for SDD flight
test

DOT&E 13-Track metrics for software
stability

DOT&E 13-Determine viability of putting
270-volt power on 28-volt signal bus

DOT&E 13-Track low observable (LO) and
non-LO repair times

DOT&E 14-Update IOT&E Schedules

DOT&E 14-Complete MDL testing before
flight test

DOT&E 14-Complete Block 2B weapon
delivery accuracy

DOT&E 14-Require contractor to do finite
element analysis on F-35B bulkhead

DOT&E 14-Resource Block 3 VSim
adequately

DOT&E 14-Accelerate joint technical data
(JTD) verification for fielded F-35Bs

DOT&E 14-Extend decontamination tests

GAO 05-Establish an executable program      O      W      W      W
consistent with best practices and DoD
policy regarding knowledge-based,
evolutionary acquisition

GAO 07-Limit annual production                            O      W
quantities to no more than 24 aircraft
per year until each variant's basic
flying qualities have been demonstrated

GAO 08-Revisit and revise the Mid-                               O
Course Risk Reduction Plan to address
concerns about testing, use of
management reserves, and manufacturing

GAO 08-Improve JSF cost estimate                                 O
reliability

GAO 09-Report to the congressional
defense committees on the risks and
mitigation strategy for use of cost
reimbursement contracts for procurement
and plans to transition to fixed price
contracts

GAO 09-Ensure contractor performs
periodic schedule risk analyses to
improve schedule and budget actions

GAO 10-Make a new, comprehensive, and
independent assessment of the costs and
schedule to complete the program,
including military construction, JSF-
related expenses in other budgets, and
life-cycle costs

GAO 10-Reassess warfighter requirements
and, if necessary, defer some
capabilities to future increments

GAO 11-Maintain future funding at
current levels

GAO 11-Establish criteria for evaluating
the F-35B's progress and make
independent reviews, allowing each
variant to proceed at its own pace

GAO 11-Conduct an independent review of
the software development and lab
accreditation processes

GAO 12-13-Restructure JSF program by
incorporating previous recommendations
from GAO 2008-11

GAO 14-Assess/identify specific
capabilities that can be delivered to
the military services to support their
respective initial operational
capabilities by July 2015

GAO 14-Assess the affordability of F-
35's current procurement plan that
reflects various assumptions about
technical progress and future funding

DoD IG 13-Ensure LM's design and
material changes are with government
concurrence

DoD IG 13-Perform process proofing of
supply chain's critical processes and
requirements flow verification

DoD IG 13-Establish independent quality
assurance organization to review
supplier processes

DoD IG 15-Evaluate open variance on F135
engine

DoD IG 15-Resolve nonconformities of
P&W's engine software quality management
systems

Recommendations                            2009   2010   2011   2012

DOT&E 05-Have data collection and
mission replay to evaluate mission
effectiveness

DOT&E 05-Test and Evaluation Master Plan    W      C
(TEMP) should identify shortfalls in
opposing force/threats

DOT&E 05-Align requirements for each
block aircraft

DOT&E 05-Develop predictive model on
engine performance after "quick dump"
fuel ingestion

DOT&E 05-Reduce fuel ingestion
vulnerability by improving fuel bladders
around inlet ducts

DOT&E 06-Update issues from recent
operational assessment

DOT&E 06-Consider doing Initial
Operational Test and Evaluation (IOT&E)
earlier with operationally
representative aircraft

DOT&E 06-Follow May 06 Defense
Acquisition Board ideas for Partner
testing

DOT&E 06-Fund adequate full-scale aerial
target to confirm operational
effectiveness

DOT&E 06-Conduct full-up, live system-
level live-fire ballistic tests on F-35
to determine vulnerability

DOT&E 07-Retain last two system
development and demonstration (SDD)
aircraft

DOT&E 07-Ensure labs are resourced to       W      W      C
execute verification strategy and to
surge

DOT&E 07-Develop metrics for                W      C
verification strategy

DOT&E 07-Develop entrance criteria for      W      C
IOT&E

DOT&E 07-Reinstate dry bay engine fire
suppression

DOT&E 07-Reinstate engine fuel ingestion
suppression liner

DOT&E 07-Add-Improve cockpit warning        W      W      W      W
lights to F-35B for ballistic damage
before vertical landing

DOT&E 07-Retain engine fueldraulics and
liquid cooling shutoff valves

DOT&E 08-Add resources to flight testing    W      W      C
in FY09-11

DOT&E 08-Explain all test changes to        W      C
DOT&E

DOT&E 08-Initiate a Test Plan Working       W      C
Group

DOT&E 08-Stabilize production and           W      W      C
deliveries of systems needed for OT&E
and training

DOT&E 08-Complete third iteration of the    W      C
TEMP

DOT&E 08-Improve verifications simulator    W      W      C
(VSim) to meet adequate verification
testing

DOT&E 08-Restore capability to minimize     W      W      W      W
fueldraulics spillage from threat-
induced damage

DOT&E 09-Focus delivery efforts on SDD      O      W      C
aircraft

DOT&E 09-By an Operational Test Review      O      C
Team, review IOT&E test plan for Block-
3 aircraft systems

DOT&E 09-Have more transparent contract     O      C
negotiations for Block-3 test aircraft

DOT&E 09-Verify, validate, and accredit     O      W      C
test labs

DOT&E 10-Assure new flight test schedule           O      W      W
is realistic

DOT&E 10-Evaluate flight test schedule,            O      W      ?
executed versus planned

DOT&E 10-Determine impact of technical             O      W      ?
issues of helmet-mounted display

DOT&E 10-Assure software integration is            O      C
in flight test

DOT&E 10-Verify/validate msn data loads            O      W      ?
(MDLs)

DOT&E 10-Redesign On-board Inert Gas               O      W      W
Generating System (OBIGGS) to maintain
oxygen levels below where fire can be
sustained

DOT&E 11-Use event-driven criteria to                     O      ?
begin flt ops

DOT&E 11-Test transonic buffeting                         O      ?

DOT&E 11-Determine impacts of late                        O      ?
structural durability testing

DOT&E 11-Improve spare/resupply for                       O      ?
flight test

DOT&E 11-Survey test plans for                            O      ?
certifications by outside government
agencies

DOT&E 12-Make corrections to Version 4                           O
of TEMP

DOT&E 12-Conduct testing on Autonomic                            O
Logistics Information System (ALIS)

DOT&E 12-Make operational test (OT)                              O
aircraft fully production-
representative

DOT&E 12-Ensure contractor meeting VSim                          O
requirements

DOT&E 12-Assure integrated VSim and MDL                          O
testing

DOT&E 12-Continue PAO shutoff valve                              O
redesign

DOT&E 12-Consider removing fueldraulic                           O
fuses

DOT&E 12-Consider keeping dry bay fire                           O
extinguisher for the Integrated Power
Pack only

DOT&E 12-Determine ballistic event                               O
survival time

DOT&E 13-Account for historical growth
of flight test

DOT&E 13-Get VSim data for SDD flight
test

DOT&E 13-Track metrics for software
stability

DOT&E 13-Determine viability of putting
270-volt power on 28-volt signal bus

DOT&E 13-Track low observable (LO) and
non-LO repair times

DOT&E 14-Update IOT&E Schedules

DOT&E 14-Complete MDL testing before
flight test

DOT&E 14-Complete Block 2B weapon
delivery accuracy

DOT&E 14-Require contractor to do finite
element analysis on F-35B bulkhead

DOT&E 14-Resource Block 3 VSim
adequately

DOT&E 14-Accelerate joint technical data
(JTD) verification for fielded F-35Bs

DOT&E 14-Extend decontamination tests

GAO 05-Establish an executable program      W      W      W      W
consistent with best practices and DoD
policy regarding knowledge-based,
evolutionary acquisition

GAO 07-Limit annual production              W      W      W      W
quantities to no more than 24 aircraft
per year until each variant's basic
flying qualities have been demonstrated

GAO 08-Revisit and revise the Mid-          W      W      W      W
Course Risk Reduction Plan to address
concerns about testing, use of
management reserves, and manufacturing

GAO 08-Improve JSF cost estimate            W      W      W      W
reliability

GAO 09-Report to the congressional          O      W      W      W
defense committees on the risks and
mitigation strategy for use of cost
reimbursement contracts for procurement
and plans to transition to fixed price
contracts

GAO 09-Ensure contractor performs           O      W      W      W
periodic schedule risk analyses to
improve schedule and budget actions

GAO 10-Make a new, comprehensive, and              O      W      W
independent assessment of the costs and
schedule to complete the program,
including military construction, JSF-
related expenses in other budgets, and
life-cycle costs

GAO 10-Reassess warfighter requirements            O      W      W
and, if necessary, defer some
capabilities to future increments

GAO 11-Maintain future funding at                         O      W
current levels

GAO 11-Establish criteria for evaluating                  O      W
the F-35B's progress and make
independent reviews, allowing each
variant to proceed at its own pace

GAO 11-Conduct an independent review of                   O      W
the software development and lab
accreditation processes

GAO 12-13-Restructure JSF program by
incorporating previous recommendations
from GAO 2008-11

GAO 14-Assess/identify specific
capabilities that can be delivered to
the military services to support their
respective initial operational
capabilities by July 2015

GAO 14-Assess the affordability of F-
35's current procurement plan that
reflects various assumptions about
technical progress and future funding

DoD IG 13-Ensure LM's design and
material changes are with government
concurrence

DoD IG 13-Perform process proofing of
supply chain's critical processes and
requirements flow verification

DoD IG 13-Establish independent quality
assurance organization to review
supplier processes

DoD IG 15-Evaluate open variance on F135
engine

DoD IG 15-Resolve nonconformities of
P&W's engine software quality management
systems

Recommendations                            2013   2014   2015

DOT&E 05-Have data collection and
mission replay to evaluate mission
effectiveness

DOT&E 05-Test and Evaluation Master Plan
(TEMP) should identify shortfalls in
opposing force/threats

DOT&E 05-Align requirements for each
block aircraft

DOT&E 05-Develop predictive model on
engine performance after "quick dump"
fuel ingestion

DOT&E 05-Reduce fuel ingestion
vulnerability by improving fuel bladders
around inlet ducts

DOT&E 06-Update issues from recent
operational assessment

DOT&E 06-Consider doing Initial
Operational Test and Evaluation (IOT&E)
earlier with operationally
representative aircraft

DOT&E 06-Follow May 06 Defense
Acquisition Board ideas for Partner
testing

DOT&E 06-Fund adequate full-scale aerial
target to confirm operational
effectiveness

DOT&E 06-Conduct full-up, live system-
level live-fire ballistic tests on F-35
to determine vulnerability

DOT&E 07-Retain last two system
development and demonstration (SDD)
aircraft

DOT&E 07-Ensure labs are resourced to
execute verification strategy and to
surge

DOT&E 07-Develop metrics for
verification strategy

DOT&E 07-Develop entrance criteria for
IOT&E

DOT&E 07-Reinstate dry bay engine fire
suppression

DOT&E 07-Reinstate engine fuel ingestion
suppression liner

DOT&E 07-Add-Improve cockpit warning        W      C
lights to F-35B for ballistic damage
before vertical landing

DOT&E 07-Retain engine fueldraulics and
liquid cooling shutoff valves

DOT&E 08-Add resources to flight testing
in FY09-11

DOT&E 08-Explain all test changes to
DOT&E

DOT&E 08-Initiate a Test Plan Working
Group

DOT&E 08-Stabilize production and
deliveries of systems needed for OT&E
and training

DOT&E 08-Complete third iteration of the
TEMP

DOT&E 08-Improve verifications simulator
(VSim) to meet adequate verification
testing

DOT&E 08-Restore capability to minimize     W      ?
fueldraulics spillage from threat-
induced damage

DOT&E 09-Focus delivery efforts on SDD
aircraft

DOT&E 09-By an Operational Test Review
Team, review IOT&E test plan for Block-
3 aircraft systems

DOT&E 09-Have more transparent contract
negotiations for Block-3 test aircraft

DOT&E 09-Verify, validate, and accredit
test labs

DOT&E 10-Assure new flight test schedule    W      ?
is realistic

DOT&E 10-Evaluate flight test schedule,
executed versus planned

DOT&E 10-Determine impact of technical
issues of helmet-mounted display

DOT&E 10-Assure software integration is
in flight test

DOT&E 10-Verify/validate msn data loads
(MDLs)

DOT&E 10-Redesign On-board Inert Gas        W      W      --
Generating System (OBIGGS) to maintain
oxygen levels below where fire can be
sustained

DOT&E 11-Use event-driven criteria to
begin flt ops

DOT&E 11-Test transonic buffeting

DOT&E 11-Determine impacts of late
structural durability testing

DOT&E 11-Improve spare/resupply for
flight test

DOT&E 11-Survey test plans for
certifications by outside government
agencies

DOT&E 12-Make corrections to Version 4      ?
of TEMP

DOT&E 12-Conduct testing on Autonomic       W      ?
Logistics Information System (ALIS)

DOT&E 12-Make operational test (OT)         C
aircraft fully production-
representative

DOT&E 12-Ensure contractor meeting VSim     W      ?
requirements

DOT&E 12-Assure integrated VSim and MDL     C
testing

DOT&E 12-Continue PAO shutoff valve         W      ?
redesign

DOT&E 12-Consider removing fueldraulic      C
fuses

DOT&E 12-Consider keeping dry bay fire      W      ?
extinguisher for the Integrated Power
Pack only

DOT&E 12-Determine ballistic event          W      ?
survival time

DOT&E 13-Account for historical growth      O      ?
of flight test

DOT&E 13-Get VSim data for SDD flight       O      ?
test

DOT&E 13-Track metrics for software         O      ?
stability

DOT&E 13-Determine viability of putting     O      C
270-volt power on 28-volt signal bus

DOT&E 13-Track low observable (LO) and      O      C
non-LO repair times

DOT&E 14-Update IOT&E Schedules                    O      --

DOT&E 14-Complete MDL testing before               O      --
flight test

DOT&E 14-Complete Block 2B weapon                  O      --
delivery accuracy

DOT&E 14-Require contractor to do finite           O      --
element analysis on F-35B bulkhead

DOT&E 14-Resource Block 3 VSim                     O      --
adequately

DOT&E 14-Accelerate joint technical data           O      --
(JTD) verification for fielded F-35Bs

DOT&E 14-Extend decontamination tests              O      --

GAO 05-Establish an executable program      W      W      --
consistent with best practices and DoD
policy regarding knowledge-based,
evolutionary acquisition

GAO 07-Limit annual production              W      W      --
quantities to no more than 24 aircraft
per year until each variant's basic
flying qualities have been demonstrated

GAO 08-Revisit and revise the Mid-          C
Course Risk Reduction Plan to address
concerns about testing, use of
management reserves, and manufacturing

GAO 08-Improve JSF cost estimate            C
reliability

GAO 09-Report to the congressional          C
defense committees on the risks and
mitigation strategy for use of cost
reimbursement contracts for procurement
and plans to transition to fixed price
contracts

GAO 09-Ensure contractor performs           C
periodic schedule risk analyses to
improve schedule and budget actions

GAO 10-Make a new, comprehensive, and       C
independent assessment of the costs and
schedule to complete the program,
including military construction, JSF-
related expenses in other budgets, and
life-cycle costs

GAO 10-Reassess warfighter requirements     C
and, if necessary, defer some
capabilities to future increments

GAO 11-Maintain future funding at           C
current levels

GAO 11-Establish criteria for evaluating    C
the F-35B's progress and make
independent reviews, allowing each
variant to proceed at its own pace

GAO 11-Conduct an independent review of     C
the software development and lab
accreditation processes

GAO 12-13-Restructure JSF program by        *
incorporating previous recommendations
from GAO 2008-11

GAO 14-Assess/identify specific                    O      --
capabilities that can be delivered to
the military services to support their
respective initial operational
capabilities by July 2015

GAO 14-Assess the affordability of F-              O      --
35's current procurement plan that
reflects various assumptions about
technical progress and future funding

DoD IG 13-Ensure LM's design and            O      C
material changes are with government
concurrence

DoD IG 13-Perform process proofing of       O      W      W
supply chain's critical processes and
requirements flow verification

DoD IG 13-Establish independent quality     O      W      W
assurance organization to review
supplier processes

DoD IG 15-Evaluate open variance on F135                  O
engine

DoD IG 15-Resolve nonconformities of                      O
P&W's engine software quality management
systems

Note. Derived and summarized from DOT&E (2005-14), USGAO (2005,
2007-14), DoDIG (2013-15), DOT&E = Director of Operational Test and
Evaluation, GAO = Government Accountability Office, IG = Inspector
General, PAO = Polyalphaolefin, TEMP = Test and Evaluation Master
Plan, IOT&E = Initial Operational Test and Evaluation, SDD = system
development and demonstration, VSim = verification simulator, ALIS =
Autonomic Logistics Information System, OT = operational test, LO =
low observable, msn = mission, fit = flight, O = open, W = working, C
= Close, ? = unknown, - = not published yet, * = summary

TABLE 3. PARTICIPANT RESPONDENT QUESTIONNAIRE TIMING

Participant #   Total Respondents    Group      Day #1

1                       1           Manager    19 Mar 15
5                       2           Manager    19 Mar 15
7                       3           Manager    19 Mar 15
8                       4           Manager    19 Mar 15
10                      5           Manager    19 Mar 15
12                      6           Manager    19 Mar 15
14                      7           Manager    19 Mar 15
15                      8           Manager    19 Mar 15
16                      9           Manager    19 Mar 15
17                     10           Manager    19 Mar 15
18                     11           Customer   20 Mar 15
19                     12           Customer   20 Mar 15
20                     13           Customer   20 Mar 15
21                     14           Customer   20 Mar 15
22                     15           Customer   21 Mar 15
23                     16           Customer   20 Mar 15
24                     17           Customer   20 Mar 15
25                     18           Customer   19 Mar 15
26                     19           Customer   21 Mar 15
27                     20           Customer   20 Mar 15
28                     21           Customer   14 Apr 15
29                     22           Customer   19 Mar 15
30                     23           Customer   23 Mar 15
31                     24           Customer   20 Mar 15
32                     25           Customer   20 Mar 15
33                     26           Customer   20 Mar 15
34                     27           Customer   21 Mar 15
35                     28           Customer   31 Mar 15
36                     29           Customer   21 Mar 15
37                     30           Customer   21 Mar 15
38                     31           Customer   21 Mar 15
39                     32           Customer   21 Mar 15
40                     33           Customer   22 Mar 15
41                     34           Customer   21 Mar 15
42                     35           Customer   26 Mar 15
43                     36           Provider   19 Mar 15
44                     37           Provider   19 Mar 15
46                     38           Provider   19 Mar 15
47                     39           Provider   19 Mar 15
48                     40           Provider   19 Mar 15
49                     41           Provider   19 Mar 15
50                     42           Provider   19 Mar 15

Participant #     Completed      Day #28    Released

1                 10 Apr 15     16 Apr 15      N/A
5                 9 Apr 15      16 Apr 15      N/A
7                21 Apr 15 *    16 Apr 15      N/A
8                 13 Apr 15     16 Apr 15      N/A
10                14 Apr 15     16 Apr 15      N/A
12                13 Apr 15     16 Apr 15      N/A
14                13 Apr 15     16 Apr 15      N/A
15                14 Apr 15     16 Apr 15      N/A
16                15 Apr 15     16 Apr 15      N/A
17                16 Apr 15     16 Apr 15      N/A
18                10 Apr 15     17 Apr 15      N/A
19                17 Apr 15     17 Apr 15      N/A
20                17 Apr 15     17 Apr 15      N/A
21                16 Apr 16     17 Apr 15      N/A
22                20 Apr 15 *   18 Apr 15      N/A
23                3 Apr 15      17 Apr 15      N/A
24                12 Apr 15     17 Apr 15      N/A
25                14 Apr 15     16 Apr 15      N/A
26                18 Apr 15     18 Apr 15      N/A
27                6 Apr 15      17 Apr 15      N/A
28                27 Apr 15     12 May 15      N/A
29                9 Apr 15      16 Apr 15      N/A
30                10 Apr 15     20 Apr 15      N/A
31                17 Apr 15     17 Apr 15      N/A
32                16 Apr 15     17 Apr 15      N/A
33                16 Apr 15     17 Apr 15      N/A
34                14 Apr 15     18 Apr 15      N/A
35                27 Apr 15     28 Apr 15      N/A
36                14 Apr 15     18 Apr 15      N/A
37                7 Apr 15      18 Apr 15      N/A
38                9 Apr 15      18 Apr 15      N/A
39                17 Apr 15     18 Apr 15      N/A
40                19 Apr 15     19 Apr 15      N/A
41                18 Apr 15     18 Apr 15      N/A
42                30 Mar 15     23 Apr 15      N/A
43                15 Apr 15     16 Apr 15   26 May 15
44                15 Apr 15     16 Apr 15   26 May 15
46                7 Apr 15      16 Apr 15   26 May 15
47                16 Apr 15     16 Apr 15   26 May 15
48                8 Apr 15      16 Apr 15   26 May 15
49                8 Apr 15      16 Apr 15   26 May 15
50                3 Apr 15      16 Apr 15   26 May 15

Note. * Late submission allowed by Dissertation Chair

TABLE 4. BREAKDOWN OF RESPONDENTS' DEMOGRAPHICS

Org:       JPO       Mil   LM   Tot   Employee:   AD    GS   Con   Tot

JPO         10       0     0    10    JPO         5     5    0     10
Mil         0        25    0    25    Mil         12    13   0     25
LM          0        0     7    7     LM          0     0    7     7
Total       10       25    7    42    Total       17    18   7     42

Org:    Education:   HS    BS   MS    Dctr        Tot

JPO     JPO          0     1    8     1           10
Mil     Mil          1     2    21    1           25
LM      LM           0     1    5     1           7
Total   Total        1     4    34    3           42

Job      Exec/                                             Function
(One)      Dir      PM      AO     Tester   Op/Trn   Tot    (Any)

JPO         6        4       0       0        0      10      JPO
Mil         0        4      15       1        5      25      Mil
LM          5        2       0       0        0       7       LM
Total#     11       10      15       1        5      42    Total %

Job
(One)    Pol/Dec    SME    Pilot     Mx

JPO        30       70       0       0
Mil        16       64      48       20
LM         29       86      14       0
Total#     19       69      33       12

Year's
Experience
(Avg +/- StD)    Requirements   Technology      Commonality

JPO (Manager)    11.1 +/-11.4   17.4 +/-10.9    6.5 +/- 6.4
Mil (Customer)   7.8 +/- 7.7    9.5 +/-9.8      4.2 +/- 4.3
LM (Provider)    17.0 +/- 9.5   20.0 +/-14.0    14.1 +/- 9.7
Total            10.1 +/- 9-4   13.2 +/- 11.5   6.4 +/- 6.8

Year's
Experience
(Avg +/- StD)    Concurrency    Partnering      Respondent

JPO (Manager)    6.3 +/- 5.3    8.8 +/- 7.1     10.0 +/- 5.8
Mil (Customer)   5.7 +/- 5.4    4.3 +/- 4.3     6.3 +/- 4.7
LM (Provider)    9.1 +/- 5.3    10.9 +/- 10.7   14.2 +/- 7.9
Total            6.4 +/- 5.4    6.4 +/- 6.7     8.5 +/- 6.3

Note. JPO = JSF Program Office, Mil = military services, LM =
Lockheed Martin, Tot = total, AD = active duty, GS = general
schedule, Con = contractor, HS = high school, BS = bachelor's, MS =
master's, Dctr = doctorate, Exec = executive, Dir = director, PM =
program manager, AO = action officer, Op = operator, Trn = Trainer,
Pol Dec = policy decider, SME = subject matter expert, Mx =
maintainer, Avg = average, StD = standard deviation

TABLE 5. RESPONDENTS' WORD QUANTITIES

                     Q1-3          Q4-6          Q7-9
Words/Answer     Requirements   Technology    Commonality

JPO (Manager)    252 +/- 158    169 +/- 94    124 +/- 57
Mil (Customer)   418 +/- 288    340 +/- 281   180 +/- 121
LM (Provider)    200 +/- 106    147 +/- 83    174 +/- 144
Total             114 +/- 98    89 +/- 104     55 +/- 49

                   Q10-12        Q13-15       Respondent
Words/Answer     Concurrency   Partnering     Total Avg

JPO (Manager)    162 +/- 94    196 +/- 184   903 +/- 451
Mil (Customer)   270 +/- 154   313 +/- 375   1520 +/- 853
LM (Provider)    157 +/- 74    104 +/- 116   782 +/- 400
Total             75 +/- 64     83 +/- 120   1250 +/- 828

Note. JPO = JSF Program Office, Mil = military services, LM =
Lockheed Martin, Q = question, Avg = average

TABLE 6. RESPONDENTS' FAMILY CODE BREAKDOWN

                                                Manager     Customer
                                                Program     Services
                                              Office (10)     (25)

Balancing Requirements Intent (BRI)               13           37
Balancing Requirements Negative (BRN)             21           54
Balancing Requirements Positive (BRP)              7           26
Balancing Requirements Recommendation (BRR)        9           30
Balancing Requirements Sub-total                  49          144
Technology Intent (TI)                             9           23
Technology Negative (TN)                          15           61
Technology Positive (TP)                           3           21
Technology Recommendation (TR)                    15           30
Harnessing Technology Sub-total                   40          134
Commonality Intent (CMI)                          14           28
Commonality Negative (CMN)                         7           39
Commonality Positive (CMP)                        19           32
Commonality Recommendation (CMR)                   9           24
Demanding Commonality Sub-total                   41          118
Concurrency Intent (CCI)                          13           31
Concurrency Negative (CCN)                        16           46
Concurrency Positive (CCP)                         9           28
Concurrency Recommendation (CCR)                  13           31
Evoking Concurrency Sub-total                     49          133
Partnering Intent (PTI)                           12           33
Partnering Negative (PTN)                         10           42
Partnering Positive (PTP)                         14           40
Partnering Recommendation (PTR)                    7           32
Encouraging Partnering Sub-total                  41          143
Cost Increase (CI)--bad                           16           23
Cost Increase Reason (CIR)--why                   22           22
Cost Decrease (CD)--good                          14           44

Cost Decrease Reason (CDR)--why                   32           32
Cost Recommendation (CR)                           2           2
Cost Sub--total                                   86          120
Schedule Lengthened (SL)--bad                     16           25
Schedule Lengthened Reason (SLR)--why             13           21
Schedule Shortened (SS)--good                      8           19
Schedule Shortened Reason (SSR)--why               4           4
Schedule Recommendation (SR)                       3           0
Schedule Sub--total                               43           69
Performance Reduced (PRD)--bad                    14           50
Performance Reduced Reason (PRR)--why              8           19
Performance Improved (PI)--good                    4           13
Performance Increased Reason (PIR)--why            3           5
Performance Recommendation (PR)                    2           5
Performance Sub--total                            31           90
Grand Totals                                      380         951

                                               Provider
                                              Contractors   Totals
                                                  (7)        (42)

Balancing Requirements Intent (BRI)               12          62
Balancing Requirements Negative (BRN)             16          91
Balancing Requirements Positive (BRP)             11          44
Balancing Requirements Recommendation (BRR)        8          47
Balancing Requirements Sub-total                  43         236
Technology Intent (TI)                             7          39
Technology Negative (TN)                          13          89
Technology Positive (TP)                           7          31
Technology Recommendation (TR)                     5          50
Harnessing Technology Sub-total                   30         204
Commonality Intent (CMI)                          14          56
Commonality Negative (CMN)                         3          49
Commonality Positive (CMP)                        12          63
Commonality Recommendation (CMR)                   4          37
Demanding Commonality Sub-total                   30         204
Concurrency Intent (CCI)                           6          50
Concurrency Negative (CCN)                         7          69
Concurrency Positive (CCP)                         8          45
Concurrency Recommendation (CCR)                   6          50
Evoking Concurrency Sub-total                     26         189
Partnering Intent (PTI)                            5          50
Partnering Negative (PTN)                          4          56
Partnering Positive (PTP)                          9          63
Partnering Recommendation (PTR)                    4          43
Encouraging Partnering Sub-total                  20         204
Cost Increase (CI)--bad                            6          45
Cost Increase Reason (CIR)--why                    8          52
Cost Decrease (CD)--good                          21          79

Cost Decrease Reason (CDR)--why                   13          77
Cost Recommendation (CR)                           0          4
Cost Sub--total                                   48         254
Schedule Lengthened (SL)--bad                     10          51
Schedule Lengthened Reason (SLR)--why              7          41
Schedule Shortened (SS)--good                      7          34
Schedule Shortened Reason (SSR)--why               0          8
Schedule Recommendation (SR)                       1          4
Schedule Sub--total                               25         137
Performance Reduced (PRD)--bad                     4          68
Performance Reduced Reason (PRR)--why              3          30
Performance Improved (PI)--good                    3          20
Performance Increased Reason (PIR)--why            1          9
Performance Recommendation (PR)                    0          7
Performance Sub--total                            11         132
Grand Totals                                      233        1564

Note. (XX) = number of respondents

FIGURE 1. PREVIOUS JOINT AIRCRAFT PROGRAMS'
INTERRELATIONSHIPS BETWEEN PILLARS OF ACQUISITION AND
ACQUISITION STRATEGY CHALLENGES

                Pillars of Acquisition

                Schedule

                [up arrow] = Lengthens
                [down arrow] = Shortens

Acquisition Strategy Challenges

Balancing       [up arrow] Drive
Requirements               consensus
                [up arrow] Rushing to meet
                           milestones

Harnessing      [up arrow] Immature
Technology                 manufacturing
                           processes
                [up arrow] Not previously
                           attempted
                [up arrow] Immature technology
                [up arrow] Mature technology or
                           commercial-off-the-shelf (D)

Demanding       [up arrow] More stakeholders
Commonality                require more time to
                           decide

Evoking         [down arrow] Development,
Concurrency                  production,
                             testing at nearly
                             the same time
                  [up arrow] Immature manufacturing
                             processes (B)

Encouraging     [up arrow] More countries sooner
International              have more say
Partnering

                Pillars of Acquisition

                Cost

                [up arrow] = Increases
                [down arrow] = Decreases

Acquisition Strategy Challenges

Balancing       [down arrow] Combine programs to
Requirements                 get one aircraft
                [up arrow] Meet multi-Services'
                           missions and operating
                           environments
                [up arrow] Mission/requirements
                           additions or changes
                [up arrow] Contradictory aerodynamic
                           requirements (C)

Harnessing      [up arrow] Not previously
Technology                 attempted
                [up arrow] Immature
                           technology

Demanding       [down arrow] One aircraft to meet
Commonality                  needs of two or more
                             Services
                [up arrow] % of common parts decreases
                           as time passes (A)

Evoking         [down arrow] Development, production,
Concurrency                  testing at nearly the
                             same time
                [up arrow] Retrofits if problems
                           arise during testing
                [up arrow] Immature technology
                [up arrow] Mature technology or
                           commercial-off-the-shelf

Encouraging     [down arrow] More countries buying
International   [down arrow] More countries sooner
Partnering

                Pillars of Acquisition

                Technical
                Performance

                [up arrow] = Keeps or Improves
                [down arrow] = Reduces

Acquisition Strategy Challenges

Balancing       [down arrow] Contradictory aerodynamic
Requirements                 requirements
                [down arrow] Concessions eventually
                             made by one Service (C)

Harnessing      [up arrow] Need for asymmetric
Technology                 advantage
                [down arrow] Not previously
                             attempted
                [down arrow] Immature
                             technology
                [up arrow] Mature technology or
                           commercial-off-the-shelf (D)

Demanding       [down arrow] As Service autonomy
Commonality                  is threatened (B)

Evoking         [down arrow] If not enough testing
Concurrency                  or the right type of
                             testing--simulated or
                             live flight
                [down arrow] Retrofits if problems
                             arise during testing (B)

Encouraging     [down arrow] Desire for coalition
International                interoperability
Partnering

(A) Balancing Requirements

(B) Harnessing Technology

(C) Demanding Commonality

(D) Evoking Concurrency

(E) Encouraging International Partnering

Note. Specific instances of strategy effect on pillars from TFX
F-111, JVX V-22, and JPATS T-6 programs were generalized from the
dissertation. Color scheme demonstrates that acquisition strategy
challenges are not mutually exclusive.

FIGURE 3. JSF PROGRAM PERCENT CHANGE IN COSTS, QUANTITIES,
AND SCHEDULE

Millions of Dollars FY 15           2001          2013

Research & Development Cost    41,283.20     62,000.10
Procurement Cost              183,154.40    273,070.70
Total Program Cost            226,354.80    338,949.60
Program Unit Cost                  78.98        137.95
Total Quantities                   2,866         2,457
Acquisition Cycle Time            116.00        237.00

Note. Adapted from General Accounting Office (2003, 2004) and
Government Accountability Office (2005-2011a, 2012a, 2013-2015),
FY = fiscal year
COPYRIGHT 2017 Defense Acquisition University Press
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Witek, Roger
Publication:Defense A R Journal
Date:Jan 1, 2017
Words:18485
Previous Article:Increasing army supply chain performance using an integrated end-to-end metrics system.
Next Article:Federally mandated furloughs: the effect on organizational commitment and organizational citizenship behavior.
Topics:

Terms of use | Privacy policy | Copyright © 2020 Farlex, Inc. | Feedback | For webmasters