Operational Testing: Redefining Industry Role.

Besal, R.E.

Whitehead, Steven K.

Contractor access does not guarantee faster, cheaper acquisition process

According to some defense contractors, a closer and more direct working relationship between the operational testers and industry is warranted and necessary. While we agree with that premise, we still do not believe, as some contractors have said, that operational testing is the "long pole in the tent" in fielding a system, and that giving industry more access to program documentation will, in some manner, significantly improve the acquisition process.

While some small increase in efficiencies might be possible, we suggest that industry look elsewhere for significant gains. Currently, Navy operational testing averages less than 1 percent of the total program cost and takes less than 7 percent of program development time (assuming five-year fielding effort). By these measures, the Navy operational test process is efficient and effective.

The Reality

The acquisition process will not be noticeably shorter or cheaper with industry observation of testing and access to documents. What is needed is a shift in the pervasive mindset within the acquisition community that:

* The war-fighter needs it now. Program managers often use this sales pitch. But what the war-fighter really needs is a system that works reliably when he needs it the most--during combat. A system that works in an unstressed, non-threatening situation is useless if it fails to perform during combat.

* We can make it right after we get it in the fleet. Providing war-fighters with a system that does not meet either their needs or requirements is a professional and moral disservice. Planning to "fix it after it's fielded" places the operational tester in a no-win situation. We are charged to evaluate system performance to the level stated in the operational requirements document. If the system can't perform to that level, it fails. If you know it won't perform to that level beforehand, make that known and work the issue out with the requirements sponsor and operational tester. It's frustrating for us (and expensive for industry) to find our after planning an operational test and expending funds that someone on the developmental side knew the system was unable to perform, but didn't say anything in time to adjust planning.

* "We can do it faster, better, and cheaper." Edward Comstock, principal assistant for acquisition, programming and budgeting in the office of the chief of naval operations said, "When a program gets down to the brass tacks, there are three principal factors: cost, schedule and performance. And the statement I often get from my program managers is, 'I can give you two."' When the best that can be achieved is two our of three of these, stop selling all three and be forthright on which one is not going to make it.

What It Will Take

If we're to achieve "faster, better and cheaper acquisition, we need more awareness within the acquisition community of the following issues.

1. Sometimes you just can't get there from here. At some point, preferably sooner than later, something may preclude a system from achieving its required capabilities. This could he cost, schedule or a limitation in current technology. When this occurs, admit the reality of the situation and concede the effort. Invest the remaining resources in areas that offer greater promise of success.

2. Some technologies need to mature before they're adaptable for military use. The inherent immaturity of leading edge technologies often makes them unsuitable for use in a military environment (for example, shipboard, foxhole, etc.). They have their place in demonstrations and experiments, but not in combat. Some are unstable, unproven, or just too cost-prohibitive for widespread military use.

3. Commercial products don't always translate easily into military environments. Off-the-shelf products were not designed for combat, but for work in your home or office. Modifying COTS can lead away from inter operability, and too often has resulted in dysfunctional [C.sup.4]1 and weapon systems. The marketing technique of proclaiming, "It's COTS, so it doesn't need testing," is irresponsible risk management.

4. A schedule can drive a program into the ground. Schedules are important, but they should nor be the driving force of a program. Schedules are tools better used to determine efficiency, rather than when a system is ready for use. System development metrics need to be based on performance or achievement, not on the calendar. (We will accept that program managers have a real challenge here, because their funding is calendar-based.)

5. Betting a program on a single test is poor risk management. Would you try to graduate from a university with a degree in engineering by taking just one comprehensive final exam at the end of your four years? Unfortunately, some program managers try a variation of this idea when they reduce or just bypass the opportunity for assessments by operational testers, and place all their chips on the line in a single operational evaluation. Our experience in this regard has reconfirmed. Hope is not a strategy for success.

6. Operational testing is the user's quality assurance process. End-of-the-line quality assurance is a poor production practice. Too often, operational testers are excluded from participating in developmental testing events and program reviews. Despite the clamor to "get the operational testing community involved early," there is significant resistance to this concept. Many of the "traditional" excuses are still heard, such as, "It's too early, and the system still has problems. ... If they see something they'll tell everyone. ... Acquisition decisions don't affect operational testing."

7. Why can't industry have access to testing documents? We would ask the same question about developmental test documents (acquisition strategies, program baseline agreements, developmental test plans, data, and reports). The operational testing director needs developmental testing and industry test plans, data, and reports to plan for efficient, non-redundant tests and to capitalize on lessons learned.

8. Knowing and following the rules is a sure way to succeed. We have a bounty of directives, regulations, instructions, policies, and procedures that have been established for acquisition and resting. The operational tester is dependent upon two of the fundamental items: an operational requirements document and a test and evaluation master plan. Both are requisite documents for any program, and both essential to conduct operational testing. A disciplined following of the guidance for acquisition and testing is critical to your success.

9. Operational testers test based on requirements, not on contract specifications. When industry is provided the specifications for a system, or the government releases a request for proposal, the operational requirements document (ORD) or concept of operations document (COOD) should also be provided. The requirements sponsor develops the ORD, and under the new Defense Department acquisition regulations, an ORD may not exist early in the program. In this case, a COOD (based on the mission needs statement) will be the only document describing (albeit at a fairly high level) how the system will be used. The user, the Navy system developer and the operational tester should develop the COOD jointly. The resource/requirements sponsor can then use the COOD to develop the ORD.

10. The key word in operational test and evaluation is "operational." In the intended environment, against the projected threat, employing typical maintainers and operators is how operational is defined. We are often asked, "How are you going to test the system?" Our response is that we will test it the way the fleet will use it, in an end-to-end mission scenario.

Navy Position

The "value added" by controlled involvement of industry in operational testing outweighs the detriments. Observations of the testing by qualified industry representatives could produce benefit, in several areas. First, the observers could put into context any problems discovered during the test. They would see firsthand what worked and what didn't. They could provide feedback to the program office as well as their company, with an insight that they previously lacked.

How they will be able to observe testing where only an operator and his system are present, such as in aircraft tests, will require further exploration.

We are currently developing a standard operating policy that will define how industry representatives will be allowed to observe our operational testing. It will include a requirement for the representative to execute a non-interference agreement, precluding any interaction with test personnel or equipment unless specifically requested by the operational test director. The observer will be allowed to take notes and will be provided data, with the program office permission, to analyze failures if they occur.

Access to documentation is another area where the benefits outweigh perceived risks. Access to the ORD is essential if industry is to understand what the war-fighter needs. The requirements--not the contract vehicles and specifications--are used by the operational testers as a measuring tool. When a mission need statement is translated to an ORD, and an ORD is translated to a contract specification, "things" can get lost.

Contractor access to the approved test and evaluation master plan (with contractual or financial information redacted) is sensible. The test and evaluation master plan is a program office document, however; and its control is their responsibility and prerogative. Access to approved operational test plans makes sense too. Our standard procedure is to offer the program manager a brief on the test plan after it has been approved, and the contractor might find benefit in attending. For some reason, our experience has been that program managers generally decline this brief.

Industry observer participation in integrated product teams is also an issue not in the control of the Navy operational testing community. Program managers charter IPTs, and they or their empowered representative chair them. We are invited participants and have no control or influence in who is allowed to attend, observe, or participate. It seems reasonable to include industry representatives to comply with Defense Department and Navy acquisition reform initiatives of partnering with industry.

With regard to providing early test data to industry, the current procedure for the Navy is to provide the program manager, as expeditiously as possible, all data relating to a system failure or anomaly discovered during operational testing. We accomplished this by sending an anomaly message from the commander of the operational test and evaluation force to the program manager. The program office restricts us from interfacing directly with industry developers. This is to prevent the possibility of perceived tasking to correct or investigate the cause of an anomaly. Direct operational tester feedback to industry developers might be misconstrued that the tester is setting a requirement for the system through informal discussions. We do not want to be placed in a position of defending a casual "It would be nice if the system could ..." remark that the developer mistakenly construes as a requirement to pass testing.

Industry, program managers and operational testers all can benefit from open communication--but we must be realistic in our expectations of improvements in quality, economy and efficiency.

Rear Adm. Robert E. Besal is the commander of the Navy Operational Test and Evaluation Force, in Norfolk, Va. Steven K. Whitehead is the organization's technical advisor.