Pay me a little now or a lot later; test and evaluation concepts to assist in managing affordable weapon systems programs.
Most important, the results from the test programs are used to determine if the product's performance will meet the warfighters' needs and are worth buying in production quantities. Well-thought-out test strategies developed in concert with other acquisition management strategies early in the program's life cycle help programs remain affordable in development and throughout their life cycle.
The Critical Role of Early Involvement
Early involvement by all stakeholders is key to program success. Getting the requirements right, translating them into contractual documents, and articulating an executable acquisition strategy are vital to developing affordable programs. Anything that can be upfront in the life of weapon system programs reduces uncertainty in the program's final outcome.
Early involvement of the test community in the process is no exception. In fact, due to the significant costs of development testing, the involvements of all members of the test and evaluation (T&E) community early in a program's life will result in a more stable and affordable test strategy. T&E specialists should be included in all phases of program development, in the following roles:
* Requirements development Integrated Product Teams (IPTs) should include T&E specialists to determine if the stated operational requirements are measureable, testable, and "make sense" when considered in an operational context.
* Contracting IPTs can use T&E experts in the development of Statements of Work (SOWS) and Request(s) for Proposals (RFPs) to ensure that T&E requirements included in these documents are complete, concise, and clear to industry and that industry has a fair opportunity to include these costs in their proposals to ensure they meet the warfighters' performance requirements.
* Source selection teams should include T&E persons on the technical review team to ensure that proposals include all necessary testing, that those costs associated with T&E have been properly identified and priced. More importantly, T&E specialists can determine if the proposed solutions can be reasonably evaluated in an operational environment to meet the needs of the warfighter or if additional testing should be added to the proposal costs to make up for a shortfall.
Early involvement of the T&E community is key to long-term program success in meeting cost, schedule, and performance goals. But only through the total integration of all aspects of T&E will testers be able support program managers in achieving affordable and executable programs for the warfighter and the taxpayer.
The Benefit of Integrated Testing
Simply stated, integrated testing is a collaborative effort by the entire T&E community to develop a strategy for test events with data that can be shared by all members of the test and program team for independent analysis, evaluation, and reporting. Efficient integrated testing should include all types of T&E: contractor developmental testing, government developmental testing, live-fire T&E, and operational testing. Both the director of operational T&E and the deputy assistant secretary of Defense for developmental test and evaluation strongly endorse integrated T&E to ensure program success.
A word of warning: While developmental testing is planned to be a "period of discovery," operational testing should be a "period of confirmation." Too often, operational testing has become a second discovery period. This occurs when there is inadequate time for developmental testing, a misunderstanding of operational concepts, or changes to the baseline configuration without full understanding of their impacts. If done properly, there are significant benefits to integrated testing:
* Integrated testing allows the sharing of data that all members of the test team can use to do their own analysis and evaluation. This reduces the number of actual test events by eliminating redundancy. Although integrated testing will never replace the statutory or prudent requirement to conduct separate and independent operational T&E, enabling operational testers to use verified data from integrated test events earlier in the program will provide clearer insight into a system's ability to complete initial operational test and evaluation (IOT&E) or operational evaluation (OPEVAL) and reduce the length of a system's operational test and the subsequent evaluation.
* Integrated testing and shared evaluation allows full visibility by the entire test and program team into the complete test program during the development phase and beyond. It ensures a smooth transition of primary responsibility for T&E throughout the life cycle, from contractor developmental testing to government developmental testers and, finally, to the operational testers.
* Integrated testing ensures that potential operational deficiencies are caught early in the development phase and corrected with enough time to verify that the fixes actually work, reducing the risk to the program's schedule and cost. Integrated testing conducted in a mission context with operational users participating will discover problems relating to operational effectiveness and suitability. These deficiencies can be identified, corrected, and verified long before the systems are formally evaluated during IOT&E or OPEVAL.
Integrated testing can enhance the affordability of a program by reducing the risk a program will experience cost growth late in development due to an unsuccessful operational test event that forces a retest, with the incumbent increase in cost and schedule. Integrating all test events will ensure a larger "bang for the buck," and help spend every T&E dollar effectively and efficiently. While many in the T&E community are accused of testing for the sake of testing, integrated testing forces the community to decide how much testing is truly required to make an informed recommendation and which test events are "nice to know." Anything that can be done to reduce the length of test events or the number of events while allowing the T&E community to gather enough data to make a complete, independent evaluation of a system's operational capabilities must be considered, to help program managers produce affordable weapon systems.
Ultimately, the true measure of success is whether a weapon system can be operated effectively by a trained warfighter, in the environment for which it is intended, against a representative enemy. Too often, traditional developmental testing processes only evaluate systems performance against specifications and leave out a mission context. While this may increase the likelihood of a successful test, losing an opportunity for early operational evaluation can create a significant residual program risk. Passing an operationally realistic test should be the new standard; units execute missions, not weapon systems. Integrated testing allows the T&E community and the warfighter earlier insight into operational performance, enhancing knowledge of the system's strengths and limitations.
A Cost-Effective Alternative to Testing
Integrated testing is a wonderful strategy to assist weapons programs in developing and executing affordable programs. Another alternative to improve affordability is a concept called design of experiments (DOE). DOE is an outstanding T&E concept that may allow a program to reduce the number of test events yet obtain the same insight into a weapon system's operational effectiveness and suitability.
DOE is a systematic method that uses quantitative, mission-oriented tools to predict how well a weapon system would perform within its operational envelope. Its objective is to uncover the most important factors to successful mission accomplishment. Testers can then systematically vary test factors to gather information using statistical measures of merit, called power and confidence. The importance of DOE is that it reduces the amount of test events required to gather the most important data with which to make sound acquisition decisions, while still ensuring test adequacy and confidence.
Reducing the amount of testing required should not only reduce cost and schedule but by collecting better scientific data earlier in the program's development phase it can foster making better decisions.
In a 2010 memorandum, "Guidance on the Use of Design of Experiments (DOE) in Operational Test and Evaluation," J. Michael Gilmore, Ph.D., director of operational test and evaluation, wrote: "The purpose is to ensure that the right type of data and enough of it are available to answer the questions of interest." Gilmore further states: "[DOE] is a structured process to identify the metrics, factors, and levels that most directly affect operational effectiveness and suitability and that should be reflected in detailed test plans."
DOE has been effectively used in the DoD operational test community since approximately 2008. It has been instrumental in saving a number weapons programs money and time, while obtaining the data the operational test community needs to evaluate each weapon system's operational effectiveness and suitability. During the quick reaction test of JDAM, the use of DOE saved approximately $3 million and 2 weeks of testing, compared with traditional testing. After a number of reliability failures, JASSM missiles were modified, and operational testing was required to validate the corrections. Traditional testing called for 21 missile shots. Using DOE, the number of required shots was reduced to 16, and the amount of information obtained was greater for fewer resources. Using DOE to test the modifications to the JASSM saved the Air Force and American taxpayer nearly $7.2 million and reduced the test period by 60 days.
Despite this success, there is no single way to apply DOE in weapon system acquisition. There must be dialogue between all members of the test team, contractors, developmental testers, and operational testers to determine its best use throughout the test program. Additionally, discussion must take place to ascertain the proper variables, factors, and levels that will be used and how those things will be defined. Ultimately, DOE should be considered for use across the entire T&E continuum.
A Well-Planned Test Program Can Be Affordable
Due to the costs of testing, T&E can have a significant positive impact on a weapon system's affordability. This may not translate into doing fewer tests, but creating a better test program can be a means of avoiding both upfront costs and the costs of redoing your mistakes. Early involvement by the T&E community in requirements can prevent ambiguous, unobtainable, or un-testable operational requirements. Early involvement in contract and program development will not provide the government with all the information needed to make decisions, but it can make sure the right test program is put on contract the first time. Poorly stated operational requirements and badly articulated contractual requirements waste time and money.
The same can be said for redundant and unnecessary repeated test events. Integrated testing allows the early discovery of deficiencies and seeks to maximize the use of valuable test resources, which will save the program office cost and schedule in the long term. Design of Experiments is a concept that, when methodically used within a program's test strategy, can reduce test assets and events, while still providing adequate data to allow the T&E community to independently assess progress or attainment of operational effectiveness and suitability.
The true success of a T&E program is an affordable, effective, and suitable weapon system in the hands of the warfighter as soon as possible. Weapon systems that fail to complete IOT&E or OPEVAL and are forced to go back through development and re-execute IOT&E not only become more expensive, but also fail to support the warfighter. The T&E community can be "value added" to any weapon systems program manager. The more emphasis program managers place on solid, integrated test and evaluation planning early in a program, the better chance that program will successfully complete IOT&E and get in the hands of our warfighters when they need it.
As the saying goes, "Pay me now, or pay me a whole lot more later!"
Bohn is a professor of acquisition management at DAU. He has 13 years of acquisition management experience and was the Operational Test Agency commander for the U.S. Marine Corps.
The author can be reached at Michael.firstname.lastname@example.org.
|Printer friendly Cite/link Email Feedback|
|Publication:||Defense AT & L|
|Date:||Sep 1, 2011|
|Previous Article:||Requirements in the affordability crosshairs.|
|Next Article:||Better metrics for better communication as a foundation for better program buying power; dialogue between program oversight and program execution...|