The challenges and opportunities of implementing human systems integration into the navy acquisitions process.

Over the last decade, the Department of Defense has placed increased emphasis on including considerations of human capabilities and limitations into systems engineering and acquisition processes. The purpose of this article is to provide an overview of how the Navy is implementing Human Systems Integration (HSI), the process of incorporating considerations, characteristics, capabilities, and limitations of human operators and maintainers within acquisition decision making at a level commensurate with decisions regarding hardware and software. More specifically, this article will address some of the policy initiatives, organizational changes, and implementation challenges of incorporating HSI into the acquisition life cycle to insure better total system performance and lower total ownership cost.

**********

In the past, incorporating human considerations into the military systems acquisition process was often overridden by the need to deliver systems to the warfighter as quickly and inexpensively as possible. In fact, there were some major opponents to the ideas of bringing human factors into the fold. Perhaps the most outspoken of the critics was Admiral Hyman Rickover, who characterized the promulgation of a human factors program into the research, development, engineering, and production in shipbuilding as "about as useful as teaching your grandmother how to suck an egg" (1970). Since that time, however, several factors have brought the need for considerations of human capabilities and limitations to light. Among these were (a) recommendations from a then-General Accounting Office (GAO) report, which cited human error rates as a key (in the range of 50 percent) factor in major system failures (GAO, 1981), (b) several very highpublicity military, industrial, and commercial aircraft accidents involving human error during the 1970s and 1980s, as well as (c) the recognition by the Department of Defense (DoD) of the impact of manpower costs on total system life cycle costs. Coupled together, these three factors have brought the necessity of considering human factors to the attention of the defense acquisition community. As a result, the consideration of human factors early in the acquisition process was mandated in the 1991 DoD5000.2 Instructions. Subsequent versions provided more and more detail on the component domains that must be considered in the acquisition process, including considerations of manpower, personnel, human factors engineering, habitability, safety, and health hazards. While the instructions have provided a springboard for inclusion of these factors into the acquisition process, the real work remains to be done to seamlessly integrate these considerations into the systems engineering framework so that human considerations have an equal footing with hardware and software considerations in the systems engineering process. Human systems integration (HSI) is the process by which this is accomplished.

The keyword in HSI is "integration," which includes integration of human considerations into the systems engineering process as well as the integration of the domains within HSI. The old adage about giving a child a hammer and everything is a nail applies here. Experts in a specific domain tend to view the solutions to human performance issues to be within their own respective domain. For example, from the point of view of a training specialist, a human performance deficit will likely be viewed as a training problem with a training solution. However, from the point of view of a human factors engineer, the same problem may be viewed as a human factors design problem. Furthermore, from the point of view of a manpower analyst, the same problem may be viewed as a problem of allocation of tasks to an operator. In reality, human performance problems may have a number of solutions from each domain or, more likely, the solution may be a combination of solutions. As this illustrates, in addition to the need to integrate with non-HSI acquisition and systems engineering domains, there is a need for integration between the HSI domains to collaborate effectively towards solutions to human performance issues within the context of cost and schedule constraints.

While the implementation of HSI is a challenge for all branches within DoD, the purpose of this article is to provide an overview of the work that the Navy specifically has undertaken on these challenges with respect to changes in policy and organization, as well as the issues of implementation that are faced by major acquisition programs in terms of organization, planning, and conducting analyses.

POLICY

The mandate for HSI within the DoD 5000 series brought considerable attention to the need for this process to be a part of the larger acquisition process at all levels. Among the biggest supporters of this effort was Admiral Vern Clark, the Chief of Naval Operations (CNO) at the time that the latest version of the 5000 series was promulgated. Under his direction, the relevant Offices of the Chief of Naval Operations (OPNAV) and the major systems commands for Surface (SYSCOMs)--Naval Sea Systems Command (NAVSEA), Aviation--Naval Air Systems Command (NAVAIR), and the Space and Naval Warfare Systems Command (SPAWAR) were tasked to develop plans to insure that HSI was a part of current acquisition programs. The current CNO, Admiral Mike Mullen, has persisted in supporting these policies and OPNAV organizations have set about the development of policy, instructions, and guidance to accommodate the emphasis on HSI including extensive work to map analyses, processes, and deliverables required for HSI to align more readily with the systems engineering and acquisition frameworks. This will insure that outputs from the HSI domains interleave with current milestones and phases in a manner that allows for a real impact on design decision-making trade-offs. Among the major Navy initiatives is the development of the Systems Engineering, Acquisition, and Personnel Integration (SEAPRINT) effort, the goals of which are to standardize Navy HSI policy, ensure HSI issues are addressed, and to facilitate HSI analyses. The SEAPRINT is a Naval Enterprise-wide approach, which includes seven actionable tenets for the implementation. These are: (a) initiating HSI early in the acquisition process, (b) identifying HSI issues and planning analyses to mitigate these issues, (c) insuring that HSI is "crosswalked" throughout relevant acquisition documentation, (d) making HSI processes a factor in source selection, (e) execution of an integrated technical process, (f) conducting proactive trade-offs within the acquisition process, and (g) conducting HSI milestone assessments. A more detailed discussion of SEAPR1NT is beyond the scope of this article. For more information on this Initiative please refer to the Navy Human Performance Center website at https://www.spider.hpc.navy.mil/index.cfm?RID=WEB OT 1001399

ORGANIZATION

The major SYSCOMs of the Navy, responsible for developing and acquiring systems to support the warfighters, each had unique challenges given large differences in organizational structures and business processes. For example, NAVSEA was able to establish an HSI directorate (NAVSEA 03) within their organization, sanctioned with technical authority to review the status of HSI performance within new acquisitions, as well as to upgrade and to modify legacy programs within an organizational structure heavily centered on specific surface activities and warfare systems.

The NAVAIR organization is based on the systems engineering competencies required to develop aircraft and related weapons systems. In other words, there is a competency for logistics, a competency for program management, a competency for science and engineering, etc. Personnel are pulled from each of these competencies directly to support any given acquisition program. As such, responsibility for specific HSI products are spread throughout the organization and ownership of these processes may fall within several competencies. The NAVAIR approach was to (a) institute HSI measures within its Systems Engineering Technical Review (SETR) process rather than develop a specific directorate charged with HSI review and, (b) to realign specific competencies related to human performance science and technology to provide expertise and support to individual programs through Integrated Product Teams (IPTs) participation.

There was a danger, however, in having each of the Navy SYSCOMs developing unique HSI processes and organizations in isolation. Without close coordination between the SYSCOMs, it was possible, in fact likely, that each would develop its own unique ways of doing business. Given the complexities of network-centric warfare, the necessities of interoperability, and the tight coupling and integration between systems required in today's warfare, these unique approaches would likely result in untenable mismatches that would have serious consequences to cost and schedules of acquisition programs, especially those where integration of air and ship operations are vital. Consequently, the Navy's SYSCOMs have worked together within a "Virtual SYSCOM" to insure that policies and processes do not diverge. Thus, representatives from the "Virtual SYSCOM" have collaborated extensively on the development of guidance for program management and development of metrics for human performance, technical (i.e., programmatic) performance, and common HSI processes. Further, the CNO provided funding for the Human Systems Performance Analysis Capability (HS-PAC) effort to develop an infrastructure to support the distribution of data relevant to human performance, thereby fostering the integration of research, development, and fleet operational activities.

EDUCATION

The implementation of HSI into the acquisition and systems engineering processes would have little likelihood of success without some effort to educate the workforce on the processes involved. As such, each of the SYSCOMs has put significant effort toward this task. Early on, these efforts focused on educating a wide range of individuals, from technical directors and program managers to systems engineers to scientists and engineers within specific HSI disciplines (e.g., Human Factors Engineering, Manpower, Personnel, Training, etc.), on basic to advanced topics in HSI. In part, as a result of these initiatives, universities and colleges are beginning to offer degrees and certificates in HSI as well. In fact, the Naval Postgraduate School now offers a graduate degree in HSI to its students.

IMPLEMENTATION

The major challenges with the implementation of HSI have been within acquisition programs themselves because, to some degree, a cultural change has been required to more fully integrate the disciplines of HSI into well established systems engineering and acquisition activities. A number of factors have come into play on how successfully HSI can be integrated into these processes, not least of which is acceptance of the value of HSI by program management and communication of this acceptance throughout the program. Other significant factors include considerations of the current acquisition phase (earlier is better to be most effective) as well as more typical concerns of funding profile, schedule constraints, and trade space (e.g., legacy equipment/new design). The following sections will discuss some of the major challenges to the development of requirements for HSI, planning to meet those requirements within the acquisition schedule, as well as conducting the analyses necessary to meet opportunities to insure high levels of human performance (and the resultant system performance) and operator situational awareness while maintaining manageable workload levels.

HUMAN PERFORMANCE REQUIREMENTS

One particularly challenging aspect of implementing HSI within an acquisition program is that the science that guides human performance is relatively new in comparison to the physical sciences, and much more subject to individual variation. Thus, the development of requirements and associated metrics that can be monitored and traced across the acquisition life cycle is difficult. This is especially true for such human constructs as situational awareness, workload, and fatigue, which have both physical and cognitive components. The challenge is in developing highlevel requirements in early requirements document that can be traced and monitored from early concept exploration and refinement to sustainment and disposal of the system. Workload, for example, can be thought of as the ratio of tasks to the time to complete them. However, it can also be considered to include such cognitive factors as frustration and cognitive effort. Thus, wide latitude of interpretation could be made with a requirement to reduce workload unless tied to a very specific and testable definition provided in the requirements.

One approach that has met with a great deal of success within the DDG 1000 (formerly DD[X]) ship program was to use manning levels as a Key Performance Parameter (KPP) in the Operational Requirements Document (ORD). In the case of this program, the threshold value of this KPP represented a significant challenge to designers that could most likely only be met through innovative design to support fewer operators performing a much broader mission, without sacrificing operator situational awareness or substantially increasing workload. However, while this manpower KPP has worked well in the case of DDG1000, it is by no means a panacea to be applied by other programs as a substitute for careful requirements and function analyses. In this particular case, the manpower KPP provided a useful tool for developing metrics to measure the impacts of design decisions against the risk of manpower increase, which may work well within a program focused on a total ship but would probably have less utility for smaller craft and/or weapons system. The bottom line is that the qualities of metrics within HSI are no different than those usually tracked within an acquisition programs. They should be chosen based on their validity, reliability, relevance to the unique issues of the mission, and should be directly tied to cost, schedule, and performance parameters of concern.

THE HSI PRODUCT TEAM

Generally, most programs have integrated HSI by implementing IPTs, or in some cases, Cross Product Team (CPT) structures, into their processes. And, as with any other IPT or CPT within a program, success is dependent on such factors as leadership, empowerment, and having the right skill mix to do the job. Perhaps at least as essential to the success of these teams is external integration with The relevant "non-HSI" disciplines. The best and brightest ideas generated within the IPT are of no value if they are not communicated effectively to program management, engineering, and design disciplines outside of the team. Thus, program management support and communication of that support throughout the program are essential to the success of HSI. If other organizations and individuals within the program view HSI as purely an academic exercise, then value related to better designed systems and decreased total ownership cost will not be likely to be realized.

THE HSI PLAN

Another key to the success is the development of a HSI plan that includes not only the goals and visions for HSI, but which also provides a process and schedule, which aligns closely to the larger program reviews, milestones, and deliverables. Without this alignment, opportunities for making a human performance impact to the design are greatly reduced, being overcome by acquisition events and design decisions already made without the benefit of HSI input. Again, the need for good communication external to the HSI IPT is important. Allied with this issue, however, is the necessity that acquisition programs allow time and funding to allow for human performance analyses and trade studies to be conducted within sufficient time to make these impacts. Thus, while it is essential that the HSI plan align with the acquisition schedule, it is also important to have the support of the program and an understanding that the dividends of designing for the sailor will pay off in total ownership cost, less rework, and higher total system (i.e., hardware, software, and human) performance.

HSI ANALYSES

Ultimately, the goal of HSI is to integrate considerations of human capabilities and limitations into the design decision-making process already being utilized for hardware and software. Integration of HSI analysis into the acquisition and systems engineering process is the key to achieving this goal. Just as it is prudent and necessary to perform analyses, testing, and verification for software and hardware integration, these same activities are required for integrating the human operator into the system. The following discussion describes some of the analysis activities that can assist in insuring that this integration takes place.

Top-Down Function Analysis (TDFA) is a family of systematic analyses and resulting documents that decompose the mission of an emerging system in a manner that links hardware, software, and human performance requirements to the intended mission (Bardine, Goff, & Wilson, 2003; Wallace, Winters, Dugger, & Lackie, 2001; Gordon, Burns, Sheehan, Ricci, & Pharmer, 2005). An excellent example of TDFA was conducted within the Multi-Mission Maritime Aircraft (MMA) program (Gordon, Burns. Ricci, & Ramsden, 2005). The primary purpose of this TDFA was to determine specific areas on which to focus training development. However, the data within this analysis have value to support human engineering decisions on what specific tasks and functions may require design support as well (e.g., decision aids, display and control, automation, etc. Further, these data could support decision making related to manpower and the allocation of tasks/functions to operators and maintainers. The utility of such an approach is that the data can be shared across HIS domains to provide a common framework from which to make engineering decisions.

The DDG1000 program has had a great deal of success in insuring a tight integration between the HSI CPT activities and system engineering activities within the program through the Mission System Design Analysis (MSDA) process. These analyses represent operators and maintainers within the context of mission performance providing a means of expressing functions and capabilities explicitly across hardware, software, and human aspects of the design (Wallace & McKneely, 2006).

Both the MMA TDFA process and the DDG1000 MSDA process are good examples of large-scale Navy programs implementing analysis techniques that provide opportunities for dialog between systems engineering and HSI disciplines. However, these activities have had equally important benefits that go beyond strengthening coordination and communication within their respective acquisition programs. The activities have also provided unique opportunities for these programs to regularly interface directly with fleet subject matter experts (SMEs). Through the interaction with individuals who are experts in the military domains, Designers and developers of the systems to support these domains have gained a better understanding of their intricacies and complexities, thereby increasing the probability that the systems that are delivered meet the true requirements of the operators and maintainers. Most HSI professionals would agree that more is better when it comes to these interactions.

As these programs have continued to progress beyond function and task analysis activities, the interaction with the fleet becomes more and more important. As such, a critical component of HSI is to conduct human factors engineering analyses focused on the usability of human system interfaces. Ideally, these activities are conducted iteratively throughout the acquisition life cycle. In the initial stages of a design concept, human factors engineers actively participate in the design process regularly conducting such activities as heuristic evaluations (i.e., usability analysis utilizing "rules of thumb" of good human factors design) and audits to insure compliance with human engineering standards and guidelines. Warfighter feedback is actively sought in these stages through activities such as focus groups where design concepts are storyboarded and presented for review and comment. During these activities, SMEs may be asked to cognitively walk through scenarios and procedures to Identify potential usability issues related to workload, errors, and situational awareness.

Like the function analytic activities described above, heuristic evaluation, Standards compliance audits, focus groups, and cognitive walkthroughs have had the benefits of allowing a better understanding of the domain, the opportunity for direct warfighter involvement in the design process and, perhaps most importantly, the opportunity to identify potential usability issues early enough in the design process, to make an impact before design changes are prohibited by cost and schedule concerns. As the design concept matures, usability testing may be conducted iteratively by using interactive prototypes and, ultimately, operational systems to continue to identify usability issues and to determine whether the issues identified represent system (i.e., hardware, software, and/or human) performance issues that must be addressed.

While warfighter interaction is essential to HSI, programs have had several challenges to taking this approach. First, active duty fleet personnel with the needed knowledge, skills, and abilities for a particular domain, especially newly conceptualized domains, are rarely in endless supply and must balance their time between their "day job" and assisting in the development of future systems. Thus, many times, there are simply not enough qualified participants available to Conduct test and assessment able to conclusively support design decisions. Second, humans-in-the-loop data are often difficult to analyze and interpret in a timely manner, especially for highly complex systems and warfare domains. Third, a number of design issues focus on extremely hazardous activities where ethical considerations would prohibit or limit the ability to conduct human in the loop evaluations.

HUMAN PERFORMANCE MODELING

Over the last decade, the Office of Naval Research (ONR) has funded a number of programs focused on human performance modeling, the representation of certain aspects of human behavior in a form that allows for simulation-based prediction (Campbell, et al., 2002), as technologies that may have the potential to mitigate some of these issues. One research thrust of the ONR-sponsored Manning Affordability Initiative (MAI) was to investigate the potential for human performance modeling applications to design teams with the ability to more rapidly conduct design trade-off activities and provide opportunities to manipulate projected operator-system interactions in an often more cost effective manner than humans-in-the-loop studies. The modeling research and development within MAI were primarily focused on investigating how human performance modeling could support human-centered design to realize manning reduction on future naval surface combatants. More specifically, the design trade space utilized for this research program centered on the development of a human-centered design console to support manning reduction within the Air Defense Warfare (ADW) suite of a combat information center. Human performance modeling efforts were conducted within this program to perform design trade-offs on console design, the flow of operator tasks, and the allocation of tasks to operators.

In addition, the effort investigated techniques for increasing the fidelity of models by verifying model predictions with data collected in a humans-in-the-loop study using current operators executing a realistic and challenging ADW scenario on prototype consoles (Scott-Nash, Carolan, Humerick, Lorenzen, & Pharmer, 2000). The results of these investigations demonstrated the potential value of Utilizing human performance modeling techniques to provide the engineers with a structured method of quantifying differences in human performance between alternative designs. Human performance modeling techniques may have the potential to help resolve some of these issues and are being embraced by the acquisition community.

Examples of human modeling techniques currently being utilized by systems designers include anthropometric, cognitive process, and task network modeling. Anthropometric modeling focuses on the physical attributes of the projected user, by replicating them into to 3-D graphical figures in order to provide systems designers with realistic ideas on how conditions, objects, and tasks associated with the planned environment may impact the human operator. There are a number of cognitive process models, which, as their name implies, function to simulate some aspect or aspects of human cognitive activity. For example, some cognitive models have been developed to function as interface evaluation tools by taking characteristics of the task and interface into account with research established on human capabilities and limitations in regard to perception, cognition, and motor processing to model interactions between the human operators and their systems (Campbell et al., 2002). These simulations can provide system designers with a great deal of insight related to human interface interaction, thus allowing for realistic performance predictions (Zachary, Campbell, Laughery, & Glenn, 1998). Timing and accuracy data associated with the planned work environment can be obtained by utilizing task network models, which are discrete event simulations based on detailed task requirement data (Laughery & Corker, 1997). If properly incorporated within the test and evaluation phase of system design, human performance models can provide assistance in both design assessment and validation.

CONCLUSION

As is hopefully evident from the discussions in the preceding sections, it is clear that the Navy has placed a great deal of emphasis on designing systems with operators and maintainers in mind. As initiatives in organizational structure, policy, process, and education continue to take hold, it is expected that the increased attention to these systems will pay dividends in terms of better total system performance at a lower total ownership cost.

REFERENCES

Bardine, D., Goff, U., & Wilson, W. (2003). TDFA methodology and tools. In: Proceedings of the 2003 Human Systems Integration Symposium of the American Society of Naval Engineers (CD-ROM).

Campbell, G.E., Holness, D.O., Buff, W., Astwood, R., Pharmer, J.A., & Commarford, P. (2002). Human modeling decision-making in the military: Technology catches up with psychology. Symposium conducted at the 110th Annual American Psychological Association Convention, Chicago, IL.

General Accounting Office. (1981). Effectiveness of U.S. forces can be increased through weapons system design (Publication No. PSAD-81-17). Washington, D.C: Author.

Gordon, J., Burns, J., Ricci, K., & Ramsden, C. (2005). Identifying training requirements early in system acquisition: HSI and the Navy's multi-mission maritime aircraft. In Proceedings of the American Sociey of Naval Engineers Human Systems Integration Symposium (CD-ROM).

Gordon, J., Burns, J., Sheehan, J. Ricci, K., & Pharmer, J. (2005) A framework For enabling HSI within systems engineering. In Proceedings of the American Society of Naval Engineers Human Systems Integration Symposium (CD-ROM).

Laughery, Jr., R., & Corker, K. (1997). Computer modeling and simulation. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (2nd ed.). (pp. 1375-1381). New York: John Wiley & Sons, Inc.

Rickover, H. (1970). Memorandum for the Chief of Naval Material. Atomic Energy Commission, Washington, DC.

Scott-Nash, S., Carolan, T., Humerick, C., Lorenzen, C., & Pharmer, J. (2000). The application of a validated human performance model to predict future military system capabilities. In Proceedings of the 2000 Interservice Industry Training Simulation and Education Conference (I/ITSEC) (CD-ROM).

Wallace, D.F., & McKneely, J.A.B. (2006). Return on investment: Lessons learned in application of HSI to DD(X) acquisition program. The Leading Edge 6(1), 9-11.

Wallace, D., Winters, J., Dugger, M., & Lackie, J. (2001). Human-centered Systems engineering process guidance. Naval Surface Warfare Center Dahlgren Division (NSWCDD) TR-01/101.

Zachary, W., Campbell, C., Laughery, R., & Glenn, E. (1998). Application of human modeling technology to the design, operation, and evaluation of complex systems. (Report No. 9800727-9705). Ft. Washington, PA: CHI Systems, Inc.

Dr. Jim Pharmer is a senior research psychologist at the Naval Air Warfare Center Training Systems Division in Orlando, FL, where he supports implementation of human systems integration tools, techniques, and methodologies into the systems engineering process for surface and aviation programs. He holds a PhD in applied experimental human factors psychology from the University of Central Florida and an MS in engineering psychology from the Florida Institute of Technology.

(E-mail address: [e-mail])