Automatic test systems: unique vs. common-core management.

Historically, individual system program offices developed and replaced unique automatic test systems (ATS) to support single weapon systems. However, significant increases in deployment footprints, parts obsolescence, and greater sustainment challenges led to a change in acquisition policy that favored "common-core" ATS that support multiple weapon systems. To date, the common-core ATS initiative has garnered little support due to a lack of substantive data supporting the expected benefits in a practical setting. Our research used a case-study approach to compare two common-core ATS programs to two weapon system-unique ATS programs. It was discovered that a lack of funding is the most critical obstacle to common-core ATS program success.

**********

From 1980 to 1992, the DoD spent over $50 billion acquiring automatic test systems (ATS). In that period, procuring unique automatic test systems to support single weapon systems was the norm. In 1994, the DoD made a dramatic change to their automatic test systems acquisition policy: common-core automatic test systems that supported multiple weapon systems were henceforth preferred over automatic test systems tailored to support a single weapon system. Expected benefits of this new policy included more reliable equipment, better supportability, less cost, smaller logistics footprint, and less manning. To date, the common-core automatic test systems initiative has garnered little support across the U.S. Air Force due to lack of substantive data supporting the expected benefits in a practical setting. Although this common-use automatic test systems policy has been in place for more than 10 years, the majority of the automatic test systems procured in the 1980-1992 "bubble" are still in service and facing severe aging and obsolescence issues.

Our research examined Air Force automatic test systems programs that differ in management approach-specifically, equipment managed as "common-core" at a separate office versus automatic test systems managed by a specific weapon system program office. Two case studies were performed: the first compares cruise missile test equipment (common-core) to intercontinental ballistic missile test equipment (unique), and the second uses a similar method to compare two test equipment systems supporting the F-15 aircraft. The research goal was to determine if the expected benefits of common-core ATS are being realized in a practical setting, and if not, to clarify common-core ATS hindrances so that senior Air Force leaders can improve the process of procuring and managing common-core ATS.

BACKGROUND

The ATS have evolved considerably over the years but their basic function has not changed: they are used to identify and isolate failed weapon system components, to facilitate component adjustments back into specifications, and to assure a system or component is ready for use (OSD [AT&L], 2005a). Automatic testing is frequently required due to the complexity of modern electronics--manually testing all components and circuit paths in typical modern systems is virtually impossible and at best impractical and time consuming.

The DoD defines an automatic test system as a "fully-integrated, computer-controlled suite of electronic test equipment hardware, software, documentation, and ancillary items designed to verify the functionality of Unit-Under-Test assemblies" (DoD ATS Executive Agent, 1997). Automatic test systems are generally comprised of three major parts, as depicted in Figure 1: automatic test equipment, test program sets, and test program set software development tools (OSD [AT&L], 2005a).

[FIGURE 1 OMITTED]

ATS PROGRAM MANAGEMENT APPROACHES

As early as the 1960s and through the early 1990s, weapon system program managers had only one option for ATS procurement: develop stovepipe ATS that supported their specific weapon system (Wynne, 2004b). Over the last decade, however, the DoD and its ATS Executive Agent have written considerable guidance to consolidate ATS development and limit unique ATS development (VandenBerg, 2004; Wynne, 2004a). The DoD's objective is to pull ATS development away from the weapon system program manager (PM) and allow a separate PM, outside the weapon system program office, to integrate the new weapon system into an existing family of common ATS (Wynne, 2004a).

WEAPON SYSTEM-SPECIFIC ATS MANAGEMENT

Under the weapons system specific approach, each system program office acquires, supports, improves, and replaces the ATS for its system independently of other programs (Wynne, 2004b). This management ideology is easy to implement but may be inefficient. This approach appears to be wasteful of resources, since multiple weapon systems will confront similar challenges, and the resulting upgrade, sustainment, and replacement programs will thereby inevitably end up funding similar, if not identical technologies (MacAulay Brown, 2002, pp. 1-2). Multiple ATS types also complicate logistics and sustainment for deploying forces because of their increased mobility footprint.

COMMON-CORE ATS MANAGEMENT

Under the common-core ATS management approach, program management is consolidated and the Service Components pursue common-core ATS to support multiple platforms (Wynne, 2004b). Common-core ATS is very difficult to implement since individual System Program Offices must adjust their schedules and requirements to accommodate the needs of several users. In turn, it significantly complicates the contractor's responsibilities and efforts to satisfy multiple user system requirements. Within the Air Force, common-core ATS are still funded by weapon system program offices that pay the common-core ATS program in proportion to the amount of ATS their weapon system program requires. This funding strategy is further complicated because weapon system PMs are concerned about paying more than their fair share for the sake of commonality (MacAulay Brown, 2002, p. 1-2). The weapon system programs are also held at greater risk if other programs responsible for paying their proportional share suffer funding cuts, because support equipment is often one of the first requirements to be cut. With these drawbacks in mind, there are benefits to common-core ATS. If designed and built correctly, common-core ATS offers efficiencies in acquisition, logistics, and sustainment (MacAulay Brown, 2002, pp. 1-2). The Air Force has attempted to enforce commonality in ATS through initiatives such as the Modular Automatic Test Equipment (MATE) program of the late 1970s and early 1980s. Such programs, however, failed to achieve success (MacAulay Brown, 2002, pp. E2-E3).

DOD ATS POLICY

Congressional language in the Fiscal Year 1993 Conference Report directed that "Comprehensive and uniform DoD-wide policy and guidance to the Acquisition and Management of Maintenance and Diagnostic ATE be developed and implemented and OSD oversight responsibility be established (Wynne, 2004b, p. 1)." Also, the Fiscal Year 1994 Appropriations Bill contained a recommendation for the Secretary of Defense to create an ATS acquisition policy (Wynne, 2004b, p. 1).

In April 1994, the Office of the Under Secretary of Defense (Acquisition, Technology & Logistics) published a memorandum stating that the DoD "components shall satisfy all acquisition needs for Automatic Test Equipment hardware and software by using designated ATS families" (Greening, 1999, p. 5). This memorandum also appointed the Navy as the DoD Executive Agent for ATS and requested a coordinated Executive Agent Charter. Since that time, the DoD's common-core ATS policy has been formally included in DoD Regulation 5000.2-R, where it remained until May of 2003. At that time, the Secretary of Defense downsized Instruction 5000.2-R from more than 200 pages to 36 pages, and in the process, removed all ATS policy references (OSD [AT&L], 2005b). Until the issue could be addressed, all Service Components continued to follow the ATS guidance in DoD Instruction 5000.2-R, Change 4 (Johnson, 2004).

On July 28, 2004, the Office of the Under Secretary of Defense (Acquisition, Technology & Logistics) reissued the latest DoD ATS policy via a memorandum stating that the policy would be included in the next issuance of DoD Instruction 5000.2-R (Wynne, 2004a). The memorandum also cancelled the Navy's role as the DoD Executive Agent. It also stipulated that ATS Service matters would be coordinated by the ATS Management Board, comprised of each Service's lead ATS office and chaired by the Navy. The ATS policy is as follows:

   To minimize the life cycle cost of providing automatic test systems
   for weapon systems support at DoD field, depot, and manufacturing
   operations, and to promote joint service automatic test systems
   interoperability, program managers shall use approved DoD ATS
   Families as the preferred choice to satisfy automatic testing
   support requirements. Commercial-off-the-Shelf solutions that comply
   with the DoD ATS Technical Architecture should only be used if
   the Milestone Decision Authority concurs that an approved DoD
   ATS Family will not satisfy the requirement. Automatic test system
   selection shall be based on a cost and benefit analysis over the
   system life cycle (Wynne, 2004a).

In September of 2004, the ATS Management Board drafted a joint memorandum of agreement, signed by each Service Acquisition Executive, detailing the processes and procedures that each Service will follow in satisfying ATS requirements (VandenBerg, 2004).

METHOD

Our first case study examined missile ATS (Ford, 2005). Cruise Missile ATS supports AGM-86B Air Launched Cruise Missiles, AGM-86C/D Conventional Air Launched Cruise Missiles, and AGM- 129 Advanced Cruise Missiles. Because Cruise Missile ATS supports more than one weapon system, it was categorized as common-core ATS for this research. The intercontinental ballistic missile (ICBM) ATS involved in this research supports only the Minuteman III weapon system and was categorized as weapon system-unique ATS.

Our second case study focused on F-15 ATS (Howe, 2005). The Avionics Intermediate Shop (AIS) supports F-15 line replaceable unit testing for airframe, engine, navigation, combat, and pilot safety systems. The AIS was categorized as common-core for our research because it is managed as such. The F-15 Tactical Electronic Warfare System Intermediate Service Station (TISS) supports only F-15 electronic warfare line replaceable units and was categorized as weapon system-unique ATS.

The article's research hypothesis is that the success of an ATS program is mainly dependent on program management, which secures adequate funding required to properly sustain and support the assigned ATS. The hypothesized model is depicted in Figure 2.

[FIGURE 2 OMITTED]

To test the hypothesis, the authors conducted 23 interviews with system program office (SPO) and depot managers, financial analysts, and equipment specialists, as well as SPO/depot engineering staff and technicians, and Major Command subject matter experts. Next, all associated documentation was reviewed, including briefings, budget reports, budget estimate submissions, and program management directives. The interviews and document reviews were designed to elicit common-core versus weapon system-unique ATS differences in four investigative areas: management differences, funding, assessments of long-term sustainability, and Major Command assessments of their field units' ability to support their assigned support equipment with the available ATS resources and SPO support. To facilitate data collection and analysis, the authors developed a data categorization matrix (Ford, 2005, Howe, 2005). This matrix satisfied several other research needs as well: it ensured a thorough data review; provided a way to discern the most meaningful data; assisted program comparisons; facilitated general theory building; and provided a logical path others can follow if this research is extended to other ATS. The quantity and quality of interview data collected within the strategic missile and F-15 case studies were considered to be equal.

After using the matrix to categorize the collected data, the authors then compared the results for each investigative area to subjectively determine the degree of difference. Next, the four investigative areas were prioritized from most different to least different, as the basis for theorizing dependencies between the four investigative areas (program management, funding, sustainment, and supportability).

RESULTS

The most significant findings by area and program are shown in the Appendix. Funding was the single greatest issue, with common-core ATS programs consistently being underfunded compared to their weapon system-unique ATS counterparts. For example, it was found that no one is quite sure whose responsibility it is to include or defend Cruise Missile ATS requirements into the Program Objective Memorandum (POM). As a result, no money has been budgeted in the POM for Cruise Missile ATS in the last seven years, despite the fact that Headquarters Air Combat Command has linked this issue to the inability to properly sustain Cruise Missile ATS beyond the 2008-2010 timeframe. Cruise Missile ATS has only received small amounts of operations and maintenance and Material Support Division funds to solve component-level problems. Tester-level problems require Element of Expense Investment Code 583 funds, which the Cruise Missile has received only once in the past seven years, and used to fund a study examining obsolescence problems.

One outcome of this funding disparity is the difference in obsolescence mitigation strategies within the Cruise Missile and ICBM programs. In the late 1990s, both programs' ATS had many of the same obsolescent components, but these problems were addressed in entirely different fashions. The Cruise Missile ATS group struggled for three years to just fund a service life extension study. After obtaining the Code 583 study funds in 2003, they opted to follow the study's suggestion of pursuing a form, fit, and function approach that addresses each obsolete component on a priority basis, hopefully extending the Cruise Missile ATS sustainability to 2030--the life of the Cruise Missile fleet. They chose this option because they did not think they would receive the funding for a new ATS; however, this approach relies heavily on the Air Force supply system to provide the necessary parts, thus putting the Cruise Missile ATS at risk when item managers dispose of parts deemed obsolete by other weapon systems. In contrast, the ICBM program was able to obtain $100 million for an entirely new ATS with first deliveries in 2005, designed to support the Minuteman III system through 2020. Furthermore, the ICBM ATS program was able to partner with industry to provide configuration control and parts replenishment rather than rely on government item managers as they had before.

The F-15 AIS and TISS ATS follow a similar funding pattern, though not as dramatic. The common-core AIS program has a wide array of unfunded requirements. AIS survey respondents related the following inadequacies in terms of recent depot funding trends: for Material Support Division Engineering, funding has been at about 30 percent of the actual requirement; for Depot 540 (software) funds, about 80 percent of the requirement, and; for Code 583 funds, about 15 percent of the actual requirement. This is in stark contrast to the equivalent figures for TISS depot sustainment funding, where respondents reported receiving all requested monies, with only one exception, during FY2005.

Overall, strategic missile and F-15 ATS findings for funding and sustainment were the most different, while program management and supportability were the most similar. The four investigative areas were stratified from most different to most similar as follows: first funding, then sustainment, followed by program management, and finally supportability. Overall, these results led to reconsideration of the hypothesized Figure 2 model.

Interviewees of common-core ATS (Cruise Missile and AIS) strongly linked funding to the ability to sustain the ATS over the long-term. In contrast, funding was not a primary concern within the unique ATS groups (ICBM and TISS), and they perceived their sustainment plans to be solid. This relationship between funding and sustainment served as our starting point for a dependency model because of their strong correlation. This correlation seems obvious, but we felt it was important to establish a firm starting point before proposing theoretical relationships between the investigative areas. Given this starting point, dependencies were posited between all four investigative areas as indicated in Figure 3.

[FIGURE 3 OMITTED]

Correlating funding with sustainment seems logical, but which is dependent on the other? There are two alternatives: 1) the funding level dictates the sustainment plan, or 2) the sustainment plan dictates the funding level. The first alternative appears to match the realities of the Air Force's fiscally constrained environment. The second alternative assumes a program office will receive all the funds required to execute an ideal program--not a realistic expectation in the authors' opinion.

Our findings indicate that program offices built their sustainment plans based on the funding they secured, not the other way around. For example, the ICBM ATS program received all the funds they required to replace all the obsolete ATS--an ideal situation. On the other hand, the Cruise Missile ATS program received limited operations and maintenance funds and a limited amount of funds from the Material Support Division to address component-level solutions over a protracted period--a risky plan with serious implications. Assuming our findings are accurate, the sustainment plan is understandably dependent on the funding level. Given this dependency, it would follow that program management's available options also depend on the funding level.

Lastly, is supportability more directly dependent on funding, the sustainment plan, or the design of program management? In the two case studies, supportability was more directly associated with the ATS sustainment plans built in the mid-1980s and with the priorities of program management. One could argue that supportability could be linked with funding as well, but we postulate that the link to funding is indirect, based on the data collected from both ATS case studies.

CONCLUSION

Our research indicates that ATS is more sustainable and supportable when managed as part of the supported weapon system. This result was expected, but for a different reason; as indicated in Figure 2, the hypothesis was that a dominant link exists from program management to the other three investigative areas. However, our research led to the development of a new dependency model (Figure 3) that is entirely different.

It was difficult to determine which management approach was more efficient because funding is only one aspect of efficiency. As the research evolved, we realized common-core ATS funding shortfalls consumed considerable time and senior leadership resources as well. We can say that funding was the most significant problem for common-core ATS in both case studies.

Future research should apply our methodology to other common-core ATS systems. Other work could also narrowly focus on the common-core ATS funding process across all Services to identify optimal funding strategies. Finally, research should identify other common-core programs and determine how well they compete in the POM process: do all common-core programs suffer the same funding problems seen in these case studies?

In conclusion, although Air Force guidance is in line with the DoD common-core ATS policy, it appears that a corresponding common-core ATS funding strategy is lacking. To realize the expected benefits of common-core ATS, a DoD funding strategy must be implemented that overcomes the current problems with common-core ATS funding. Three potential strategies include: 1) common programs are fully funded by the responsible agency, 2) weapon systems proportionally pay for their required support, and 3) Air Force corporate structure is modified to provide better "care and feeding" for common programs. Until a strategy is implemented, Services and program offices will continue to work around the DoD policy, maintaining control and funding responsibility within their system program offices. Although more costly for DoD, this approach currently exposes programs to less risk and ensures an effective ATS system within their control.

APPENDIX RESEARCH FINDINGS

INVESTIGATIVE AREA: PROGRAM MANAGEMENT.

* Strategic Missile, Program Management Differences?

--Similar organization and functional expertise

--Systems knowledge higher in ICBM group

** Cruise Missile ATS split from Cruise Missiles

--ATS SPO suffers from lack of Cruise Missile experience

--Difficult for them to link ATS impacts to weapon system and gain support of senior leadership

--Big difference in addressing obsolescence

*** Cruise Missile ATS = Form, Fit, Function Plan

*** ICBM ATS = Complete ATS replacement

* F-15, Program Management Differences?

--Lead MAJCOM liaison present at the SPO; works alongside TISS managers

--Planning impacted by position on the product life--cycle timeline; impact of Electronic Systems Test Set

--Obsolescence more urgent concern for Avionics Intermediate Shop (AIS)

--TISS program benefits from extensive community website and use of a secure classified server for software downloads

INVESTIGATIVE AREA: FUNDING.

* Strategic Missile, Funding Differences?

--FY 01-FY 05, Cruise Missiles ATS only received limited (60% of requirements) Operations & Maintenance funds and no procurement funding

--FY 02-FY 06, ICBM ATS received required Operations & Maintenance funds and ~ $107M procurement funds to replace obsolete ATS

* F-15, Funding Differences?

--TISS program received approval in 2004 POM for a $41.6M technology insertion program to replace obsolescent commercial-off-the-shelf equipment and reduce logistical footprint

--AIS program has not yet secured funding to address obsolescence concerns for the sustainment of the Antenna Test Station and Enhanced Aircraft Radar Test Station through 2025

INVESTIGATIVE AREA: SUSTAINMENT.

* Strategic Missile, ATS Sustainment Differences?

--Cruise Missile ATS group

** Sustainment plan = Form, Fit, Function replacements

--9 priorities, each addressed individually over next 25 years

** Frustrated with sustainability progress; funding = largest hurdle

** General Officer involvement

--Cruise Missile ATS = Air Combat Command/Director of Logistics (ACC/LG) first sustainment issue in 2003

--Warner Robins Air Logistics Center/Commander (WR-ALC/CC) supports ACC's concerns

--Air Force Materiel Command/Director of Requirements (AFMC/ DR) agreed they had responsibility, but cannot support funding because of expected high cost

--Air Force General Officer Nuclear Surety Steering Group monitors Cruise Missile ATS

--Cruise Missile sustainability funding issue being prepared for Air Force Board

--ICBM ATS group

** Sustainment plan = ATS replacement

--Ground Minuteman Automatic Test System (GMATS) was designed to last through 2020

--GMATS will not be supported by AF Item Managers

*** Partnered with Boeing to provide Integrated Contractor/ Logistics Support

*** Sustainment issues appeared to never go higher than the O-6 level and continue to be worked at Senior NCO level

* F-15, ATS Sustainability Differences?

--Obsolescence and diminishing manufacturing sources are the primary issues plaguing the two F-15 test equipment programs

--The AIS program faces a greater severity of obsolescence challenges

--Funding is generally considered to be the primary hurdle to overcome in the timely resolution of obsolescence and sustainment issues

INVESTIGATIVE AREA: SUPPORTABILITY.

* Strategic Missile, ATS Supportability Differences?

--Both sets of ATS suffer equally regarding supportability issues because of their equal age and design

--Notable differences

** HQ ACC assigned a functional expert to the Cruise Missile ATS System Program Offices as a "liaison"

** Cruise Missile group has closer relationship with Item Managers

** Cruise Missile group readily cannibalizes depot warehoused equipment

** ICBM group relied on depot Precision Measurement Equipment Laboratory vs. Item Managers

* F-15, ATS Supportability Differences?

--Overall characterization of field units' abilities to maintain TISS and AIS was positive

REFERENCES

Department of Defense (1996, March 15). Defense Acquisitions. DoD Directive 5000.1. Washington: U.S. Government Printing Office.

DoD Automatic Test Systems (ATS) Executive Agent. (1997, February 6). Memorandum of agreement among component acquisition executives. Patuxent River, MD: Naval Air Systems Command, PMA-260.

Fletcher, O. R. (1998). DoD automatic test systems handbook. Patuxent River, MD: Naval Air Systems Command, PMA--260.

Fletcher, O. R. (1998). DoD automatic test systems handbook: Major ATS components. Patuxent River, MD: Naval Air Systems Command, PMA-260, p. 7.

Ford, W. C. (2005). Comparing management approaches for automatic test systems: A strategic missile case study. Master's thesis (AFIT/GLM/ENS/05M), Graduate School of Engineering and Management, Air Force Institute of Technology (AU), Wright--Patterson AFB, OH.

Greening, M. A. (1999a). DoD automatic test systems architecture guide. Patuxent River, MD: Naval Air Systems Command, PMA--260.

Greening, M. A. (1999b). DoD automatic test systems masterplan. Patuxent River, MD: Naval Air Systems Command, PMA-260.

Greening, M. A. (1999c). DoD automatic test systems selection process guide. Patuxent River, MD: Naval Air Systems Command, PMA-260.

Howe, J. (2005). Evaluating management strategies for automated test equipment (ATE): An F-15 case study. Master's thesis (AFIT/GLM/ENS/05M), Graduate School of Engineering and Management, Air Force Institute of Technology (AU), Wright-Patterson AFB, OH.

Johnson, J. S. (2004, April 2). Automated test system (ATS) division overview. PowerPoint presentation at the meeting among component acquisition executives. Warner Robins AFB, GA.

MacAulay Brown Inc. (2002, September 30). Airforce common automatic test (AFCATE) equipment study. Submitted to WR-ALC/LEA: Warner Robins AFB, GA.

The Office of the Secretary of Defense (Acquisition, Technology and Logistics)(OSD [AT&L), Logistics Plans and Programs (LPP). (2005a). Welcome to the home page of the DoD automatic test systems executive directorate. Retrieved September 13, 2005, from http://www.acq.osd.mil/ats/default.html

The Office of the Secretary of Defense (Acquisition, Technology and Logistics)(OSD [AT&L]), Logistics Plans and Programs (LPP). (2005b). DoD Policy Relating to ATS. Retrieved September 13, 2005, from http://www.acq.osd.mil/ats/atspolcy. htm

The Office of the Assistant Secretary of the Air Force for Acquisition (Communications, Computers and Support Systems) (SAF/AQK). (1994, July 19). Automatic test systems and equipment. AFPD 63-2. Pentagon, Washington, D.C.: author.

VandenBerg, T. M. (2004, July 13). DoD Automatic test systems information brief for the JTEG. PowerPoint presentation at the meeting with Joint Test Evaluation Group, Patuxent River, MD: Naval Air Systems Command.

Wynne, M. W. (2004a, July 28). DoD policy for automatic test equipment Memorandum for Assistant Secretary of the Army (Acquisition, Logistics, and Technology), Assistant Secretary of the Navy (Research, Development and Acquisition, Assistant Secretary of the Air Force (Acquisition). Washington, D.C.: author.

Wynne, M. W. (2004b, July 28). DoD automatic test systems background Attachment to DoD Policy for Automatic Test Equipment. Washington, D.C.: author.

Capt William Ford, USAF, Capt Jeremy Howe, USAF, Alan Johnson, Ph.D., and Mai John Bell, USAF

Captain William Ford is the Executive Officer for the F-22 System Program Office, Wright-Patterson Air Force Base (AFB), which is responsible for development, acquisition, fielding, testing, and support of the F-22 Raptor. While at Wright-Patterson AFB, he attended the Air Force Institute of Technology (AFIT) where he completed an M.S. in Logistics Management. Ford has more than 19 years of experience in missile and munitions maintenance.

(E-mail address: [e-mail])

Captain Jeremy Howe is a maintenance officer assigned to the Air Force Logistics Management Agency, Maxwell AFB Gunter Annex, Alabama. He holds an M.S. in logistics management from the Air Force Institute of Technology (AFIT), and a Bachelors of Architecture from the University of Notre Dame. He was previously assigned as a supervisor of cruise missile and munitions maintenance at Fairchild AFB, Washington.

(E-mail address: [e-mail])

Alan W. Johnson is an Associate Professor of Logistics Management at the Air Force Institute of Technology (AFIT). He received his Ph.D. in Industrial and Systems Engineering from Virginia Tech and an M.S. in Systems Management from AFIT. His research interests include reliability and maintainability and their effects on weapon system life cycle management, and issues related to strategic airlift mobility.

(E-mail address: [e-mail])

Major John Bell is an assistant professor at the Air Force Institute of Technology (AFIT) and is a fifteen-year veteran of the U.S. Air Force. He holds a B.S. from the Air Force Academy, an M.S. degree from AFIT, and a Ph.D. in Management from Auburn University. His research

interests include logistics and supply chain management issues. Bell has also published in Computers & Operations Research, The Air Force Journal of Logistics, and National Defense.

(E-mail address: [e-mail])