Joint capabilities and System-of-Systems solutions.

Recognizing the need to succeed in a new multilateral, asymmetric threat environment, the Department of Defense has promoted a radical transformation in operations to promote agility and enhance responsiveness. The transformation process, as well as the resulting new order of operations, relies heavily on System-of-Systems (SOS) solutions to effectively bridge existing gaps in operations. To date, a pervasive and possibly detrimental assumption has dominated the program management arena: single-system level management tools and methods apply equally well to the acquisition of SOS solutions. This research questions the general assumption that single-system methods are effective in an SOS arena. Based on an empirical analysis by subject matter experts, this research begins to flesh out an analytical framework for understanding the resource requirements that will underpin SOS solutions for joint capabilities.

**********

The need for joint capabilities has rekindled interest in integration and interoperability-based solutions. As such, System-of-Systems (1) (SOS) solutions represent a new and important commodity class in the acquisition domain (Krygiel, 1999). In terms of the investment resources allocated to them and the operational value of the capabilities they provide, the resource requirements have tremendous implications for Department of Defense (DoD) performance. Consequently, exploration into the drivers of cost and risk is the subject of inquiry. To date, a pervasive, and possibly detrimental, assumption has dominated the program management arena: single-system-level management tools and methods apply equally well to the acquisition of SOS solutions. This research questions the general assumption that single-system methods are effective in an SOS arena. Taking the position that the field as a whole lacks adequate understanding of the unique cost drivers that influence SOS initiatives, the following research uncovers many of the drivers that influence the cost, schedule, and performance of SOS efforts.

This paper begins with an overview of the genesis of joint capabilities and then characterizes the likely implications joint capabilities will have on the acquisition of SOS solutions. Begging the question "What are the critical hurdles to achieving SOS cost, schedule, and performance requirements?" we relate the survey findings of subject matter experts (SMEs) on their perceptions of the cost drivers that influence SOS initiatives. Finally, we close with a discussion on the implications of the SMEs' perceptions on acquisition cost and risk mitigation.

THE GENESIS OF JOINT CAPABILITIES

Under the auspices of Secretary Donald Rumsfeld, DoD transformation became a compelling objective in the aftermath of September 11, 2001 (Rumsfeld, 2003). During the cold war era, military strategy was predicated on the belief that deterrence was best achieved through arms superiority. As emphasized by Wilson (1985), organization matters. The arms race was achieved by heavy reliance on scientific management principles as an organizing paradigm (Hughes, 1998). Economies of scale were achieved in arms production through a capital-intensive industrial base that stressed the principles of scientific management: hierarchy, division of work, functional specialization, and the separation of planning from operations. These strategies gave rise to a plethora of individual subcultures with distinct missions, goals, and vocabularies.

From a resource perspective, programs were defined by each Service and collectively submitted to the Office of the Secretary of Defense (OSD) for review and approval, and then incorporated into the President's budget (DoD Directive [DoDD] 7045.14). Fiscal guidance issued by the OSD at the beginning of the cycle gave each of the Services a target that reflected an equitable distribution of resources. Generally, equities were preserved and programs would get their start without a great deal of joint scrutiny.

After achieving budgetary approval, the Services would mobilize teams of SMEs who were responsible for conducting trade-off analyses and ultimately defining a given system's requirements. Eventually, the requirements were captured in an Operational Requirements Document (ORD). Adhering to the divide-and-conquer perspective of scientific management, the process was fairly linear with distinct hand-offs occurring between the actors of the various stages of system development. In the relatively stable environment of the cold war, this sequential requirements-acquisition process was not without problems, but generally satisfactory. Adversaries engaged in the arms race philosophy and adopted similar processes, thus leveling the playing field.

Unless a program was designated as joint, joint requirements were rarely explicitly considered (General Accounting Office [GAO], 2003a; 2003b). For the most part, joint programs became de facto single-service efforts (GAO, 1998). This was achieved through an assignment process that designated a Lead Service, which tended to view joint requirements through its Service lens. The scientific management strategies promoted rapid production by minimizing the turbulence that often arises from interdependent activities.

The demise of the Soviet Union, and the resulting proliferation of multiple non-state-affiliated threats, called into question the arms race philosophy as a deterrent. Additionally, the broad scale use of scientific management as an organizing principle became suspect (Chairman of the Joint Chiefs of Staff, 1996). In this multilateral-threat world, the deterrent value of massive armed forces eroded and the normative framework that defined the cold war and the DoD imploded.

The resulting small, yet pernicious, interventions were resilient to the deterrence that the arms race provided. Instead, the operational advantage shifted from a focus on mass and firepower to one of agility and precision (Chairman of the Joint Chiefs of Staff, 2000). Quite suddenly, agile and tightly integrated joint operations were needed, in which functional specialists are brought together to provide a specific capability suited to a particular operational context.

From a transparency and accountability perspective, the scientific management method of organizing activities simplified the budgeting process and facilitated oversight. But they did so at the expense of the integration and agility that is needed to deter current threats. In this post-cold war context, the scientific management philosophy became less useful. In this environment, where complexity and variability dominate, increasing scale often resulted in nonlinear diseconomies. The lessons of September 11, Operation Desert Storm, Operation Enduring Freedom, and Operation Iraqi Freedom illustrated the need to break from the scientific management guidelines and pursue organizational strategies that promoted functional integration and cross fertilization, hence, joint capabilities were desired.

THE LONG ARM OF JOINT CAPABILITIES

Recognizing that success demanded that joint capabilities pervade every aspect of DoD operations, no single unit was spared from Secretary Rumsfeld's call for transformation. Performance goals now stress adaptive planning, accelerated acquisition cycles built on spiral development, output-based management, and a reformed analytic support agenda (Rumsfeld, 2003). As stated by Rumsfeld (2003): "Instead of building plans, operations, and doctrine around individual military systems as often occurred in the past, henceforth the Department will explicitly link acquisition strategy to future joint concepts in order to provide the capabilities necessary to execute future operations" (p. 10). Highlighting its importance, Lieutenant General Yakovac recently underscored the necessity for acquisition to be interdependent to meet the needs of joint forces (Williams, 2004). Apparently, transformation means making a clean break from the practice of separate but equal.

Three specific goals of the transformation to joint capabilities are likely to impact the acquisition process substantially: 1) integrated architectures, 2) evolutionary and spiral implementation methods, and 3) best-of-breed competition. Integrated architectures are expected to provide the blueprint for where and how operations will intersect and overlap in order to provide joint capabilities (P. Wolfowitz, personal communication, August 29, 2002). First, integrated architectures will be the mechanisms to achieve multidimensional or holistic integration. Integrated architectures will require acquisition activities to integrate operations cross-organizationally from a materiel perspective, and they will demand acquisition to provide solutions that reach across the breadth of the Doctrine, Operations, Training, Leadership, Personnel, and Facilities (DOT-LPF) spectrum (P. Wolfowitz, personal communication, August 29, 2002). Second, the need for ongoing, rapid, and often times unanticipated, deployment requires the use of lean, evolutionary and spiral implementation methods (P. Wolfowitz, personal communication, August 29, 2002). Time to market is likely to take on a new level of importance and place new demands on the acquisition process. Third, competition is seen as an effective means for maintaining a best-of-breed military (Rumsfeld, 2003). Among the many goals of the transformation, best-of-breed competition stresses the need to promote competition among programs in an effort to achieve innovation, continuous improvement, and superiority. The implications these three goals have for the acquisition community are unprecedented.

Because acquisition transforms goals and decisions into reality, it is the locus where concepts become solidified into real-world tasks and operations. As such, we see at least four implications for the acquisition process.

1. The clean division of labor that insulated acquisition efforts in the past will have to be breached. Rapid deployment needs will not allow the time needed to clarify all ambiguities prior to acquisition. Not only is the acquisition arena not immune from the complexity of joint capabilities, but it is precisely within this arena that we would expect many of the hurdles over requirements and features prioritization to be fought (Slate, 2002).

2. The expanded scope of multidimensional integration requirements is synonymous with an expanded trade-space. The quality attributes of a system, especially an SOS, interact. Performance impacts modifiability. Availability impacts safety. Security impacts performance. Everything affects cost and so forth (Bass & Kazman, 1996). Not only is there no principled method for characterizing the interactions among quality attributes, but their value will vary by setting (Kazman, Carriere, & Woods, 2000). The SOS efforts, by nature of its expansiveness, will complicate the search for mutually acceptable solutions that meet joint requirements. In fact, we expect that the struggle over feature trade-offs, and their accompanying zero sum implications, will bleed over into the acquisition process. For all intents and purposes, SOS efforts should aggravate the struggle over competing desires.

3. Evolutionary acquisition lacks clarity and thus makes the search process all the more dynamic and porous (Sylvester & Ferrara, 2003). Evolutionary and spiral acquisition models further the need for acquirers to assume a greater role in the requirements process and requirers to assume a greater role in the acquisition process (Slate, 2002). Slate predicts that established organizational relationships will be altered, and such shifts almost always lead to conflict.

4. The competitive dimension will do little to arrest the struggle. Under adverse conditions, these struggles are likely to express themselves in scope creep, schedule delays, and performance shortfalls.

Even if only a portion of the above scenario rings true, it appears that the acquisition process will be held captive by the effectiveness of the feedback loops that are established. Feedback loops will be needed to clarify the ambiguity and reduce the friction of the interdependencies. More specifically, what feedback loops are required? What resource demands will they incur? By examining the critical hurdles to achieving SOS cost, schedule, and performance requirements, the survey results discussed below provide some insight into these important questions.

RESEARCH METHODS

Given that the research goals were largely exploratory, as opposed to predictive, the methods employed were qualitative in nature and a purposeful sampling technique was employed. Hence, the methods did not allow generalization to a wider population. Nonetheless, they were instrumental in providing insight into some expert perceptions in both the systems integration and cost estimation arenas. The survey sample consisted of attendees at the 2004 Systems Software Technology Conference (SSTC). Because of the type of attendees that typically participate in the SSTC, the venue was deemed relevant for capturing important insights on issues surrounding SOS.

The mission of SSTC is to provide information and training on software engineering issues and technologies. Its members derive primarily from the DoD, Army, Marine Corps, Navy, Air Force, and Defense Information Systems Agency (DISA). The SSTC is touted as being the premier software technology conference relating to the DoD. The conference focuses on matching problems to solutions as representatives from industry, government, and academia present their ideas and solutions through tutorials and presentations. Topics discussed during the week-long conference included interoperability, greater use of software engineering, architecture, reuse of designs and code, and other methods and improvements. The survey items were derived from the literature and personal interviews regarding cost drivers for SOS (see Appendix for a list of the survey items).

In total, 28 attendees visited the SSTC research booth and completed the SSTC survey (see Figure 1). The data were analyzed in two fashions. First, descriptive and correlation statistics were obtained to give insight on respondent perceptions of the factors that influence SOS cost, schedule, and performance. Based on an average majority viewpoint, the descriptive and correlation statistics provided an indicator of the relative importance for each factor (or variable) under study. The second analysis departed from the traditional approach to statistical analysis and used a grouping technique referred to as "Q Methodology." Under this approach, a factor analytical technique (2) was performed to isolate the number of statistically significant factions (or groups) that exists on a given point-of-view. Instead of demonstrating which variable had the greatest consensus, the methodology illustrated the extent to which the respondents, in their total viewpoint, shared similar perspectives and impressions on the factors that contribute to cost, schedule, and performance shortfalls. Hence, the method provided a means for assessing the extent to which similarity exists among perceptions on all of the various survey items. Analyzing the data in both fashions was particularly useful when a multitude of viewpoints existed and had to be considered. As demonstrated below, each of the statistical approaches cast a slightly different perspective on the matter at hand.

SURVEY RESULTS

For the most part, the majority of the respondents indicated extensive hands-on experience in SOS efforts. Sixty percent of the survey respondents identified SOS integration as their primary role in SOS efforts. Thirty-five percent indicated their primary role as SOS acquisition, and 5 percent indicated SOS policy. The average experience level in years ranged from 4 to 30 years with a mean of 12. Overwhelmingly, the respondents identified three critical variables as major hurdles to achieving cost schedule and performance goals for SOS efforts: 1) 75 percent of the sample identified leadership, standards, and requirements communication as major challenges (Table 1); 2) funding was seen as a major obstacle by 68 percent of the respondents; and 3) roughly 50 percent indicated knowledge, skills, and abilities (KSAs), aligning system interdependencies, and end-to-end mission threads as the greatest challenges to SOS efforts. Approximately 35 percent of the sample indicated conflict management, changing environmental demands, and information access as important determinants of difficulty.

Interestingly, correlation coefficients indicated some differences in perceptions based on years of experience in the SOS arena. The interdisciplinary element of the program, leadership, communications, and configuration management, proved to be more taxing for those with fewer years of experience than those who were more senior (p < .05, .04, .02, .04, and .00, respectively). Alternatively, the more senior estimators were more inclined to indicate coordination and logistics as a more critical element than those with fewer years of experience (p < .05).

In addition, the analysis revealed four significantly different groups of perceptions regarding the factors that stymie SOS efforts (p <.05, all eigenvalues exceeded 2.0--a statistic measuring the total amount of variation in the matrix explained by the factor) (see Figure 1). Group A indicated that requirements communication created the greatest obstacle. Information shortfalls, facilities, and information hoarding also presented challenges, albeit they were more minor in nature. Alternatively, collocation, organizational incentives, and individual goals did not prove to be an issue for this group of respondents.

Group B indicated that KSAs, collocation, schedule conflicts, and funding posed the greatest impediments to SOS. Coordination and logistics, fear, changing environmental demands, and facilities posed only minor obstacles. Conflict and negotiation, team conflict, semantic and syntactic differences, and cultural differences did not appear to create problems for this group.

Group C perceived changing environmental demands and information hoarding as major obstacles, and individual goals and collocation as minor challenges. Organizational alignment, doctrine, and fear did not prove to be contributors to failure for this group.

Finally, Group D indicated team conflict and interdisciplinary teams as major hurdles with end-to-end mission threads and KSAs posing minor obstacles. Collocation and funding did not prove problematic for Group D.

RESEARCH IMPLICATIONS

Four points appear particularly salient concerning the survey of the SSTC attendees. First, the importance of leadership, system standards, and system requirements as major challenges to SOS rings clear. The vast majority of the sample was unanimous in their beliefs that cost, schedule, and performance shortfalls were largely attributed to inadequacies in leadership, system standards, and system requirements. Hence, cost models that are capable of relating these variables to programmatic cost may prove useful to program managers (PMs). Moreover, PMs should look to management strategies that are capable of mitigating the risks associated with leadership, system standards, and system requirements in a joint capabilities context.

Second, analyzing the responses by the two different approaches was instrumental in revealing two very dissimilar, yet complementary, sets of findings. The first and more traditional approach allowed the examination of perceptions in an aggregate form and suggests the need for management levers that respond to the majority view. The second segmented the respondents in light of how similar they are to each other from a holistic viewpoint. While similar in their thinking regarding the importance of leadership, standards, and requirements, the group splintered into four different groups in terms of their similarity of perceptions across all the survey items. One perspective chose information and requirements (Group A) as the major challenge, another personnel skills and logistics (Group B), a third environmental flux and information hoarding (Group C), and the fourth team conflict (Group D). Assuming that perception of difficulties is likely to drive behavioral activities, the lack of coherence on the perceived importance of the various items has both positive and negative implications for SOS efforts. If this diversity is manifested in a SOS program, it might provide the advantage of understanding a wider range of potential problems that should be monitored and mitigated through managerial actions. Alternatively, the diversity in opinions could splinter and fragment the group and result in conflict over how to best allocate limited resources. For example, Group B is likely to emphasize resource investments in collocation to mitigate problems; whereas, Groups A and D are likely to play down the importance of collocation as a strategy for investment. Hence, managers should he on the lookout for tremendous conflict over how to proceed and where to invest resources.

Examining SOS activities from this alternative viewpoint revealed insights into the types of managerial practices that yield success. Given the fact that these efforts are typically multidisciplinary, and may be predisposed to disjointed perceptions, the need to examine SOS efforts from a group perspective may prove especially conducive to revealing important subtleties. Understanding the degree of alignment or disjointedness (i.e., using techniques that are capable of making the degree of consensus explicit), could prove beneficial to mitigating risk and delivering programs on schedule and within budget at the proper performance level. The results clearly suggest that the solution space is multidimensional and, as such, may require multipath strategies.

The third implication relates to the resource requirements that will be needed in this new arena. For the most part, cost functions have not captured transactions costs adequately because, prior to joint capabilities, transaction costs were fairly minimal. According to this sample, transaction costs (e.g., coordination costs, communication costs, and negotiation costs) are likely to demand substantial investments in time and resources.

Finally, the last implication underscores the complexity associated with searching for the critical determinants that drive SOS efforts. In terms of future research, uncertainty remains over the subtleties of the important factors. For example, what is it about KSAs that influence success? What skills are deficient? How are they deficient? What needs to be done to correct the deficiency? While this qualitative analysis revealed some important themes, it also underscored the field's lack of convergence regarding the factors that drive success and failure. We can expect this lack of convergence to complicate the search for managerial practices that can be leveraged to insure success. There is little doubt of the need for continued examination of the factors that influence SOS, managerial activities, and the feedback loops that equate with success. Without it, the very success of joint capabilities, and thus today's warfighters, will be severely hampered.

DISCLAIMER

The views expressed in this article are the authors' and should not be construed as representing the Department of Defense, the Office of the Secretary of Defense, the Director, Program Analysis and Evaluation, or the Chairman of the Cost Analysis Improvement Group.

APPENDIX

WebQ
Cost and Schedule Drivers for System of Systems Integration (SoSI)
Effort WebQ by Peter Schmolck: www.rz.unibw-muenchen.del/~p4lbsmk/
gmethod/webq Thank You For Your Participation

+4   (a) Too few here,       -4   -3   -2   -1   0   +1   +2   +3   +4
     need 2 more

+3   (a) Too few here,       -4   -3   -2   -1   0   +1   +2   +3   +4
     need 3 more

+2   (a) Too few here,       -4   -3   -2   -1   0   +1   +2   +3   +4
     need 4 more

+1   (a) Too few here,       -4   -3   -2   -1   0   +1   +2   +3   +4
     need 5 more

0    (c) Please reassign     -4   -3   -2   -1   0   +1   +2   +3   +4
     28 item(s)

     21. Funding             **   **   **   **   *   **   **   **   **

     6. Organizational       **   **   **   **   *   **   **   **   **
     alignment

     18. Personal            **   **   **   **   *   **   **   **   **
     commitmant

     22. Interdisciplinary   **   **   **   **   *   **   **   **   **
     teams

     16. Short-term vs.      **   **   **   **   *   **   **   **   **
     long-term individual
     goals

     15. Aligning system     **   **   **   **   *   **   **   **   **
     interdependencies

     31. Information         **   **   **   **   *   **   **   **   **
     shortfalls

     29. Collocation and     **   **   **   **   *   **   **   **   **
     logistics

     14. Common standards    **   **   **   **   *   **   **   **   **
     and architectures

     25. Semantic and        **   **   **   **   *   **   **   **   **
     syntactical
     differences

     33. Information         **   **   **   **   *   **   **   **   **
     hoarding

     5. Individual           **   **   **   **   *   **   **   **   **
     organizational
     incentive structures

     13. Configuration       **   **   **   **   *   **   **   **   **
     management

     27. Deconflicting       **   **   **   **   *   **   **   **   **
     schedules

     11. Leadership          **   **   **   **   *   **   **   **   **

     24. Understanding       **   **   **   **   *   **   **   **   **
     scope

     23. Team conflicts      **   **   **   **   *   **   **   **   **

     26. Requirements        **   **   **   **   *   **   **   **   **
     communication

     8. Knowledge, skills,   **   **   **   **   *   **   **   **   **
     and abilities
     of the personnel

     7. Quality of           **   **   **   **   *   **   **   **   **
     training for SOS

     1. Existing policy      **   **   **   **   *   **   **   **   **
     and procedural
     requirements of the
     personnel

     9. Interdisciplinary    **   **   **   **   *   **   **   **   **
     training programs

     28. Elicit advice and   **   **   **   **   *   **   **   **   **
     support without fear
     of reprisal

     19. Collocation of      **   **   **   **   *   **   **   **   **
     personnel

     4. Changing             **   **   **   **   *   **   **   **   **
     environmental demands
     of the various
     personnel

     3. Doctrine             **   **   **   **   *   **   **   **   **

     2. Cultural             **   **   **   **   *   **   **   **   **
     differences among the
     personnel

     20. Adequate            **   **   **   **   *   **   **   **   **
     facilities

     10. Aligning end-to-    **   **   **   **   *   **   **   **   **
     end mission threads

+1   (a) Too few here,       -4   -3   -2   -1   *   +1   +2   +3   +4
     need 5 more

+2   (a) Too few here,       -4   -3   -2   -1   *   +1   +2   +3   +4
     need 4 more

+3   (a) Too few here,       -4   -3   -2   -1   *   +1   +2   +3   +4
     need 3 more

+4   (a) Too few here,       -4   -3   -2   -1   *   +1   +2   +3   +4
     need 2 more

(a) Empty a lot

(b) Occupied a lot

(c) Too many a lot

(+4) Most Impact on Cost/Schedule/Performance

(-4) Least Impact on Cost/Schedule/Performance

TABLE 1:
SSTC SURVEY FREQUENCIES ON ITEMS DEEMED AS MAJOR HURDLES
(N=28)

Table 1: SSTC Survey Frequencies on Items Deemed as Major Hurdles

Survey Item                          N    Percent

Leadership                           21     75%
Requirements Communication           21     75
Standards                            20     71
Funding                              19     68
Knowledge Skills and Abilities       15     54
Aligning System Interdependencies    14     50
End to End Mission Threads           3      46
Configuration Management             1      39
Changing Environmental Demands       9      32
Information Access                   9      32
Organizational Alignment             8      29
Commitment                           7      25
Understanding Scope                  7      25
Deconflicting Schedules              7      25
Doctrine                             6      21
Interdisciplinary Teams              6      21
Conflict Negotiation                 6      21
Info Shortfalls                      6      21%
Semantic/syntactic Differences       5      18
Information Hoarding                 5      18
Cultural Differences                 4      14
Organizational Incentives            4      14
Command and Control                  4      14
Collocation                          4      14
Facilities                           4      14
Team Conflicts                       4      14
Coordination/Logistics               3      11
Policy and Procedural Requirements   2      7
Changing Environmental Demands       2      7
Short-Term Long-Term Goals           2      7
Elicit Advice                        2      7
Quality Training                     1      4
Interdisciplinary Training           1      4

FIGURE 1. SSTC CLUSTERS OF PERCEPTIONS ON DRIVERS OF COST,
SCHEDULE, AND PERFORMANCE SHORTFALLS IN SOS EFFORTS
(N = 28)

         Group A                 Group B

Major    * Requirements/         * KSA **
Hurdle   Communication **        * Collocation **
                                 * Schedule
                                   Conflicts **
                                 * Funding **

Minor    * Information           * Coordination
Hurdle     Shortfalls **         * Fear **
         * Facilities            * Changing Req **
         * Information           * Facilities **
           Hoarding **

No       * Collocation **        * Conflict/
Hurdle   * Incentives **           Negotiations **
         * Individual Goals **   * Team Conflict **
                                 * Semantics **
                                 * Cultural
                                   Differences **

         Group C              Group D

Major    * Changing Req **    * Team Conflict **
Hurdle   * Information        * Interdisciplinary
           Hoarding **          Teams **

Minor    * Individual Goals   * Alignment **
Hurdle   * Collocation **     * KSA

No       * Org Alignment **   * Collocation **
Hurdle   * Doctrine           * Funding **
         * Fear of
           Reprisal **

All clusters achieve an Eigenvalue of at least 2
All variables achieve a minimum p <.05 ** = p <.01

ENDNOTES

(1.) For the purposes of this research, system-of-systems does not relate to a particular implementation method as much as it does a broad class of integration and interoperability strategies. See http://www.infoed.com/Open/PAPERS/systems.htm for a discussion of system-of-systems.

(2.) See www.Qmethod.org for an overview of the methodology.

REFERENCES

Bass, L., & Kazman, R. (1996). Architecture-based development (CMU/SEI-99-TR7), Software Engineering Institute Technical Report. Pittsburgh, PA: Carnegie Mellon University.

Chairman of the Joint Chiefs of Staff. (1996, June 10). Joint Vision 2010. Retrieved August 15, 2004, from http://www.dtic.mil/jv2010/jvpub.htm

Chairman of the Joint Chiefs of Staff. (2000, May 24). Joint Vision 2020. Retrieved August 15, 2004, from http://www.dtic.mil/jointvision/jvpub2.htm

Department of Defense. (2003, November 2). The planning, programming, and budgeting system (PPBS) (DoD Directive 7045.14). Washington, DC: Author.

General Accounting Office. (1998, March). Joint military operations: Weaknesses in DoD's process for certifying C4I systems' interoperability. (GAO/NSIAD-98-73). Washington, DC: Author.

General Accounting Office. (2003a, January). Major management challenges and program risks: Department of Defense (GAO 03-98). Washington, DC: Author.

General Accounting Office. (2003b, May). Military readiness lingering training and equipment issues hamper air support of ground forces (GAO 03-505). Washington, DC: Author.

Hughes, T. P. (1998). Rescuing Prometheus: Four monumental projects that changed the modern world. New York: Random House.

Kazman, R., Carriere, S. J., & Woods, S. G. (2000). Toward a discipline of scenario-based architectural engineering. Annals of Software Engineering, 9, 5-33.

Krygiel, A. (1999). Behind the Wizard's Curtain: An integration environment for a system of systems. Retrieved January 13, 2003, from http://www.dodccrp.org/ html/pubs_pdf.html BWC

Rumsfeld, D. (2003, April). Transformation planning guide. Retrieved August 15, 2004, from http://www.defenselink.mil/brac/docs/transformationplanningapr03.pdf

Slate, A. R. (2002, May-June). Evolutionary acquisition: Breaking the mold--New possibilities from a changed perspective. Program Manager, 31(167), 6-15.

Sylvester, R. K., & Ferrara, J. A. (2003, Winter). Conflict and ambiguity: Implementing evolutionary acquisition. Acquisition Review Quarterly. 10(1), 10-15.

Williams, M. (2004, January/February). New military Deputy LTG Joseph L. Yakovac, Jr. plans to put people first. Army AL&T, pp. 74-77.

AUTHOR BIOGRAPHY

Mary Maureen Brown (D.P.A. University of Georgia) is an associate professor at the University of North Carolina. Brown's applied and scholarly activities focus on the use of information-based technologies for re-engineering operations to improve service delivery throughout all levels of government. Currently, she is serving as a lead investigator in an Office of the Secretary of Defense-sponsored research effort of the costs and risks of systems integration and interoperability. (E-mail address: [e-mail])

Robert M. Flowe is an Operations Research Analyst in the Weapon Systems Cost Analysis Division in the Office of the Secretary of Defense, Program Analysis and Evaluation, Deputy for Resource Analysis. Prior to joining OSD/PA&E, he served as Professor for Systems Acquisition Management at Defense Acquisition University, where he taught certification courses in Program Management and Systems Engineering, and served as the Course Manager for the Intermediate Software Acquisition Management course. Mr. Flowe obtained a B.S. in Aerospace Engineering from Virginia Tech in 1984 and a M.S. in Software Systems Management from the Air Force Institute of Technology in 1994. Mr. Flowe retired from the United States Air Force in 2003. (E-mail address: [e-mail])