Capability test methodology and Join Battlespace Dynamic Deconfliction.An important event for the testing and acquisition communities took place Aug. 4 to 8, 2008. The Joint Test and Evaluation Methodology, a chartered joint test and evaluation (T&E) project, conducted a test in conjunction with the Joint Battlespace Dynamic Deconfliction event, held by the Combined Test Organization for Future Combat Systems (FCS). JBD2 served as a "use case" to evaluate the effectiveness and suitability of the JTEM-developed capability test methodology.
JTEM was directed to develop, test, and evaluate methods and processes for defining and using a live, virtual, and constructive distributed environment (LVC-DE) to evaluate system of systems performance and to understand system contributions as they pertain to joint mission effectiveness. JTEM's entire CTM methods and processes, specifically the CTM measures framework, were utilized for the first time within JBD2 to demonstrate how to accomplish system of systems testing. This use of the CTM allowed testers to evaluate system of systems and the contribution of specific systems through the measures framework levels of evaluation--joint mission, joint task, and system/system of systems attribute levels. It also served as a successful proof of concept for the validation of new systems and system of systems testing methods and processes that enable comprehensive evaluation of joint capabilities.
The CTM is a formalization of existing test processes, with refinement, for designing a test of new capabilities or system of systems in a complex joint environment. The CTM is designed to augment, not replace, existing test and evaluation methods and processes, and it takes into account the unique aspects of testing joint, networked systems in a LVC-DE. Before we discuss the test, some background information is needed to understand the significance of the event.
Improving the Testing Process
Over the past several years, the Department of Defense has initiated several steps to make changes to the testing and acquisition processes that enable testing of joint capabilities in an operationally realistic joint mission environment. As DoD moves away from traditional single-system T&E approaches to the more complex system of systems approaches, the department must be able to demonstrate that future systems are fully integrated, fully interoperable, and able to meet the complexities and demands of future battlespace environments.
DoD testing and acquisition instructions are being reviewed to include changes that declare the need to test joint capabilities in a joint mission environment. The Joint Capabilities Integration and Development System (JCIDS) institutes a capabilities-based approach to identifying current and future gaps in DoD's ability to carry out joint warfighting missions and functions. In support of these policy directives, the DoD director for operational test and evaluation developed the Testing in a Joint Environment Roadmap as a call for action to establish a framework for life cycle evaluation of systems and system of systems in a joint mission environment, and to institutionalize evaluation in a joint operational context. The roadmap aims to place testing in a joint environment and joint interoperability testing at the core of T&E activity while promoting changes in how DoD does business in the areas of policy, infrastructure, and methods and processes. Policy changes are being driven through the roadmap governing body and through established DoD processes. Infrastructure networks and middleware issues are being addressed through the Joint Mission Environment Test Capability (JMETC) program under the Test Resource Management Center, a field activity reporting to the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. The JTEM joint test and evaluation project, chartered by the DoD director for operational test and evaluation in February 2006, is addressing methods and processes for evaluating capabilities in the joint environment. The August 2008 test event was part of the JTEM project.
Developing CTM Version 3.0
For the past two years, JTEM developed, tested, and evaluated methods and processes for defining and using an LVC-DE to evaluate system of systems performance and joint mission effectiveness. Before August 2008, JTEM conducted rock drills, a gap analysis, and a test event to gather feedback on its CTM development, leading to the release of CTM version 2.0 for the JBD2 test event. The FCS Combined Test Organization planned the JBD2 event to establish a rigorous test context that would allow the test participants to examine FCS test technology requirements needed for testing in a joint environment in support of acquisition milestone test activities. JTEM leveraged the event to test and evaluate CTM methods and processes when used by a typical test organization under operationally realistic conditions. For JTEM, the test article was the CTM. Results gathered throughout the planning, execution, and post-execution phases of JBD2-provided areas for CTM improvement to be incorporated into CTM version 3.0, scheduled for release at the conclusion of the JTEM joint T&E project in April 2009.
FCS and JTEM formed a natural partnership for the August 2008 test event. FCS leaders recognized the event presented challenges requiring new testing capabilities (system of systems on a large scale; move, shoot, and communicate simultaneously; seamless integration with joint elements for network-centric operations). To accommodate this new complexity, DoD strategic planning guidance demanded creation of a joint environment testing capability. As pointed out earlier, this DoD-level requirement led to the creation of JMETC and JTEM as part of the larger testing in a joint environment initiative. As the FCS testing strategy matured along with JTEM, a mutually beneficial relationship emerged and grew into the JBD2 partnership. JTEM understood JBD2 would provide the opportunity to:
* Identify issues in integrating the end-to-end CTM into existing test activities
* Develop and mature the LVC-DE
* Investigate data requirements issues
* Identify deficiencies in implementation of the live, virtual, and constructive joint mission environment.
FCS saw the partnership with JTEM as a way to establish a rigorous test context in which to examine FCS test technology requirements needed for testing in a joint environment in support of Milestone C test activities, as well as enable risk reduction for critical FCS technology areas. Equally important, JMETC used JBD2 as an opportunity to characterize the network infrastructure and mature the baseline capability required to support system of systems-level testing across the DoD components. Overall, JBD2 provided a requirement for a high-fidelity, real-time, rapidly configurable, distributed network connecting virtual and constructive models with live systems. This architecture offered the participating organizations an opportunity to test selected initiatives in an operational environment, enhancing the overall joint context of the test while allowing stakeholders to investigate these important new technologies. At the same time, participation in the test event provided the JTEM test team close-in visibility on implementation of the CTM in a joint mission environment. The overarching test goals were:
* Evaluate the effectiveness and suitability of the JTEM CTM
* Test the FCS network, technologies, and distributed environment for use in future tests for acquisition milestone decisions
* Mature the JMETC baseline capability to support system of systems-level testing across the Services.
A Complex Test Scenario
To achieve these goals, JBD2 established a complex joint mission environment composed of 16 test sites and more than 40 unique live, virtual, and constructive systems connected across four time zones. These test sites represented all four Services and the U.S. Joint Forces Command. Of these 16 JBD2 sites, 10 were reused from two previous test venues that were a part of a series of events culminating in JBD2. Seven Service and joint initiatives were included as part of the test architecture. JBD2 truly provided joint context and stressed the boundaries of a live, virtual, and constructive joint mission environment.
The JBD2 test scenario focused on the complex and dynamic battlespace deconfliction problem. The joint mission environment centered on this problem, with mission tasks selected that would require complete coordination of the Services to execute. Four unique Joint Fires and two Joint Close Air Support key mission tasks were agreed upon in order to fully stress the notional joint air ground system of systems (JAGS) under test. The scenario design called for these mission tasks to occur simultaneously and/or sequentially as the battle unfolded alongside organic fires-and-effects missions and air operations. Test design factorials incorporating materiel and non-materiel solutions were developed that required four unique test configurations for every key mission task. A "free play" environment was established in which any number of key mission task types could execute in parallel for a single test configuration.
As the JBD2 test scenario unfolded and the key mission tasks were executed, JTEM's CTM measures framework was implemented, which enabled critical test data to be collected. The data permitted evaluation of the notional JAGS not only at the attribute level but also at the joint task, joint mission desired effect, and joint mission end state/objectives levels, thus effectively describing whether or not JAGS, as a joint capability, provided the means and ways to perform a set of tasks to achieve the set of desired effects that lead to mission success. However, for JBD2 the notional JAGS test date was not the JTEM focus. JTEM's focus was on the effectiveness and suitability of the CTM to collect the right data and evaluate it up to the mission measure of effectiveness level.
A Successful Test Event
JBD2 was a major success in several ways. JTEM's CTM was fully exercised within a test environment possessing a level of complexity not seen in previous test events. The number of test sites connected, along with the number of systems represented, is extraordinary given the planning process was completed in less than one year. Additionally, the level of effort and use of the CTM were significant given the test team's short timeframe for planning and executing the test event. Reusability of previous JTEM test products and processes--Joint Fires mission threads, the JMETC virtual private network, DoD Architecture Framework products, and CTM products--exceeded expectations. Data collection and reduction were conducted at a speed, accuracy, and fidelity that set new standards for distributed testing. Eighty percent of the measures framework metrics was collected and reduced within a week of the test event. Data for all key mission task measures of effectiveness, task measures of performance, and measures of system of systems attributes were collected and reduced within a month of test execution completion.
JBD2 was clearly a successful proof of concept in collecting data and evaluating mission measures of effectiveness, according to JTEM's CTM measures framework. The data showed the impact of system of systems configurations on mission-desired effect and joint mission effectiveness. This is a significant milestone because testers can now move forward in their attempts to institutionalize testing in a joint mission environment across DoD.
From JTEM's perspective, JBD2 execution of CTM products was more than sufficient to assess the effectiveness and suitability of the CTM. The significant lessons learned from the test event were of great value to the JTEM test team as the team enhances its methods and processes and incorporates improvements into CTM version 3.0. About two-thirds of the CTM products produced during the test aligned completely or partially with the CTM guides and model descriptions. Additionally, the JBD2 test team developed several test products that were outside the domain of the CTM. JTEM is carefully considering these test products for inclusion in CTM version 3.0.
Insights on Improvement
JBD2 provided several insights that the test team could use in CTM development or in recommendations for improvements to current test and acquisition practices. A high-level summary of these insights finds:
* More focus should be spent on defining and documenting a coherent evaluation strategy and joint mission effectiveness requirements. A preponderance of early test activities and resources are devoted to environment design and build. Focusing first on the evaluation strategy will reduce the expenditure of resources at execution time.
* There is a need for a standard lexicon for conducting T&E in an LVC-DE.
* JCIDS products are not yet sufficiently available to effectively create the joint operational context for a test.
* There is a need for data standardization and improved data access and retrieval methods.
* There is a need for enterprise-level support, expertise, and coordination in order to effectively test in a joint mission environment.
Along with these insights, more detailed and specific improvements to the CTM have been captured by the JTEM team as a result of CTM usage in JBD2. These improvements will be included in CTM version 3.0, which will contain a CTM disc with navigation slide and three publications:
* The Action Officer Handbook for Testing in a Joint Environment
* The Program Management Handbook for Testing in a Joint Environment
* The Joint Mission Effectiveness Analysis Handbook These publications incorporate CTM guides that provide a roadmap and context to methods and processes for testing in a joint environment.
The Action Officer Handbook, written for typical test and event planning staffs, will address concepts supporting testing in a joint environment and will contain comprehensive CTM user guides for each step of the CTM process, printable checklist extracts to be used along with the CTM guides, and annexes containing useful information such as the CTM lexicon. The Program Management Handbook will be an executive-level publication providing an enterprise view of DoD acquisition transformation, addressing testing in a joint environment, and offering an overview of the CTM for program and capability portfolio managers. The Analysis Handbook, written for test planners and test analysts, will provide the framework and supporting information for the CTM evaluation thread process and will offer recommended analysis tools and techniques as well as annexes with supporting information and hyperlinks between sections for online use. That CTM package will prepare the program manager and test organization staff to effectively test as the capabilities-based approach to acquisition requires.
JBD2 provided the opportunity to identify issues in integrating the end-to-end CTM into an existing test activity, and it allowed the T&E community to gain a better understanding of what is required to fully realize a sufficient capability to test in a joint mission environment. The results are promising and indicate the CTM:
* Reduces cycle time for capability development and testing
* Increases the speed of data collection, reduction, analysis, and evaluation
* Integrates developmental and operational testing
* Provides a needed operational evaluation framework from which to test system of systems in a joint mission environment.
There is still more work to be done. In the near future, CTM version 3.0 must be integrated into existing test processes within the acquisition community in a real test event to inform an acquisition decision. This will further validate the CTM and provide valuable lessons learned for its improvement. This action will also serve to operationalize the CTM, providing a springboard for the CTM to be institutionalized as the methods and processes of choice for testing in a joint mission environment.
The CTM promises to improve how we test and acquire joint capabilities. This promise will not be fulfilled unless we all support the CTM as a means of providing improved T&E of joint capabilities. As testing in the joint environment continues to provide more challenges for the T&E community, the CTM will ensure that we test with relevance, thus supporting the acquisition and delivery of "plug and fight" joint warfighting capabilities.
Hutchison is the test and evaluation executive with the Defense Information Systems Agency. Lorenzo is the director for the Joint Test and Evaluation Methodology, Joint Test and Evaluation Project. Bryan is a senior analyst with Scientific Research Corporation serving as the JTEM outreach coordinator.
The authors welcome questions and comments and can be contacted at firstname.lastname@example.org, email@example.com, and firstname.lastname@example.org.