Simulation-Based Acquisition Only a 'Slogan,' Says Survey.

A Pentagon-funded survey of major defense programs revealed what many had suspected all along: that simulation-based acquisition is more of a myth than a reality.

Simulation-based acquisition is a buzzword that illustrates the notion that weapon systems can be designed and developed entirely as digital models and that those models are accepted as valid prototypes.

The sponsor of the survey was James F. O'Bryon, deputy director of operational test and evaluation at the office of the secretary of defense. The study covered 22 major defense acquisition programs. Each program office was asked whether it used simulation-based acquisition methods and, if so, to explain how it was implemented. The only program that did not provide data was the Joint Strike Fighter, because it's still in a competition, so any information on JSF models was considered proprietary. Ironically, the JSF program is among those that most heavily have used modeling and simulation in the design and development phases.

Among the questions asked were how digital models are organized, who pays for the models and whether the models are validated and accredited.

"To ensure candor from the program managers, we agreed to not release information on specific programs, but rather service-wide," said O'Bryon in an interview.

The survey, he said, "paints a rather bleak picture."

Even though the Defense Department spends millions of dollars each year developing digital models for various applications, simulation-based acquisition is not pursued in any organized manner, O'Bryon said. "We really don't have SBA. A lot of money is spent on models and simulation. But it's very disjointed. In some cases, it's spent multiple times on similar models," he said. "Most acquisition programs do not have modeling and simulation plans."

Industry executives either are being disingenuous or are fooling themselves when they say that SBA is "here and now," said O'Bryon.

During an August conference on testing and training in Orlando, Fla., the president of Lockheed Martin Missiles and Fire Control, Stan Arthur, said that simulation-based acquisition "has arrived."

That is false, said O'Bryon. "SBA is here in name. It's only a slogan ... A bumper sticker. ... It does not exist." Many people in the industry, he added, "want to declare success. It makes the industry people look good, because they don't spend money testing" the systems.

At the Defense Department, he said, "We are using simulation. But we don't have a simulation-based acquisition system, using simulation systematically. If that were the case, O'Bryon said, huge financial savings could be made.

"We are spending billions of dollars on models and it's not very well organized.

... On the training side, they are more organized than on the acquisition side." Models are used, for example, to replicate the terrain where a weapon system will operate, the atmospheric conditions, the radar signature, among others.

According to the survey, more than 80 percent of the models used in the 22 programs were unique to that project, and not shared with others. In many cases, the models are owned by contractors, nor by the government.

The problem, said O'Bryon, is that program managers don't have an obvious motivation to invest in modeling and simulation. Most of them are in the job for two or three years, which means that, by the time the models and simulation work is completed, they have moved on to other assignments. "Program managers have a high turnover rate, so they don't want to invest that money, because it will not benefit them in the short term," he said.

Sometimes, program managers prefer to not have realistic models, because they can "make the program look worse," said O'Bryon. The performance of a weapon can deteriorate when factors such as turbulence, terrain or targets are factored in. If the models are "too realistic," said O'Bryon, "the estimates look worse.

Rarely do models help a program "look better," he added. "Realistic models are nor always viewed as a positive thing."

Reform Plan

O'Bryon's office has submitted a proposal to the undersecretary of defense for acquisition, technology and logistics, which would help promote the use of SBA. The plan is called MASTER (modeling and simulation, test and evaluation reform).

The plan is built on the premise that there is "virtually no new money" for modeling and simulation. The funds would have to be taken from other areas within program budgets. The program managers, said O'Bryon, "control most of the money that could be spent on modeling and simulation."

O'Bryon is proposing the creation of a "modeling and simulation consortium," which would be funded by "negotiable arrangements with program managers, who would contribute to the consortium." The consortium would help the program managers advance their projects through the use of SBA and would encourage them to share models, when possible.

"Program managers typically need modeling support," said O'Bryon. Most frequently, he said, they need models to simulate missile or aircraft aerodynamics, atmospheric conditions, target signatures and vulnerability.

Under the current way of doing business, he said, the program manager typically pays for models that already have been paid for by someone else. "The consortium would help save money and duplication," said O'Bryon.

The government agency that would oversee the consortium, he said, is the Defense Modeling and Simulation Office. "DMSO would maintain and chair the consortium," said O'Bryon. Currently, "DMSO can't do its job because the program managers control the money. There needs to be a revision of the acquisition policy."

O'Bryon, however, does not advocate the use of modeling and simulation as a substitute for live testing. "Modeling and simulation [technology] has been sold as a cost saver. We can't substitute models for testing, because we don't know how to do models yet."

The current acquisition regulations require a modeling and simulation plan for every program, he said. "The money in SBA today is in stovepipes. They don't share" the technology. "They have a disincentive to share, because it's their neck. It's not their fault. It's the system."

O'Bryon noted that there is a law--which is not often enforced--that mandates congressional approval for any weapon system-related model that costs more than $50,000. "It's not being complied with," he said. "They are not tracking the law."

Currently, there are three pilot programs in place that will be used to determine whether SBA can work: the Air Force T-38 trainer aircraft avionics upgrade, the Army's common missile and the Navy extended range guided munition.

Frank Cappuccio, a Lockheed Martin vice president who works on the Joint Strike Fighter program, agreed that there is "a lot of rhetoric" about simulation-based acquisition. However, he said, "the savings are there."

During an SBA conference sponsored by the National Defense Industrial Association, Cappuccio said that the Joint Strike Fighter program achieved a 50 percent savings in "engineering, acquisition life-cycle time and cost." The savings, he said, "were incorporated into our [JSF] proposal bid."

The implementation of SBA, he cautioned, is a process with "numerous minefields." The biggest obstacle, he said, comes from the financial department. "SBA will test everyone's resolve," said Cappuccio. "If you have the resolve, you can convince your finance people to pass the savings to the government."

For SBA to work, he said, "it takes a consistent vision from the government." In the JSF program, that was possible because the program manager, Marine Maj. Gen. Michael Hough, is an advocate of the use of simulation and made sure it was implemented.

"A good vision is not sufficient," said Cappuccio. "You need pressure for change. ... This is something you have to do to be competitive."

Government leaders must decide, he said, whether the replacement of hardware testing with simulations ever will be accepted.

"Which company will take the first step to tell the government to not shoot an aircraft with a 20 mm cannon, that I am going to do it with modeling and simulation?" Cappuccio asked the audience at the conference. "It's not enough for program managers to champion SBA."

In the JSF program, both contractors--Lockheed Martin and Boeing--had an incentive to use modeling and simulation to cut costs, because it would make their proposals more competitive. But unless there is a competition, said Cappuccio, there are "no financial rewards for industry to cut costs. ... In something that is not a competition, it would be a smaller bill and higher risk."

For SBA to take hold, he said, "the government has to be a partner" and accept that models can offer legitimate testing tools.

Another issue the government must address, he said, is how to protect industry's proprietary data. "There is no trust in our industry," he said, because "government people give away the data in competitions."

There is an "entrenched belief in the financial community that proprietary data is what makes a company unique," said Cappuccio. The JSF school of thought is different, because "we give up proprietary data" to the government.

It's an emotional issue to those who believe that if the government takes proprietary data, "it is going to take away from the identity of the company." However, "we do it for national security reasons."

In those programs that require interoperability, "it would make sense to develop a standard for SBA. Industry would rally to it quickly," said Cappuccio. "We have done it before, for avionics standards."

The 1553 data bus standard was developed, he noted, "because every company was fed up with incompatibility. ... When it's in your best financial interest to share data and have common standards, you'll pull it off."

Philip W. Cheney, Raytheon vice president of engineering, said that SBA does not offer any significant return on investment. "It's a continuing squeeze on company resources," he told the conference. "SBA implementation is unique within each service, and at times, within individual program offices." The defense industry, he said, "lacks the ability and incentive to share proprietary data."

Air Force Lt. Gen. Leslie Kenne, commander of the Electronic Systems Center, said during the conference that she is a "zealot" about SBA. "If we don't really invest in it, we are not going to get where we need to be."

Predicting Network Performance Can Save Millions

Sandra I. Erwin

Organizations looking to invest millions of dollars in a new communications network would be wise to develop a digital model of the system before they commit to major hardware purchases. Although this advice seems common sense, an expert said, it is surprising how many companies and government agencies agree to buy new networks without really knowing if they will work properly and meet their needs.

"When you are spending hundreds of millions of dollars on a system, spending a small percentage on a simulation can help avoid making a large mistake," said Robert F. Brammer, chief technology office at TASC, a division of Northrop Grumman Corp.

The company specializes in government and defense-related information systems and analysis services. It has spent corporate research dollars recently to come up with modeling technologies to simulate communications networks and signals processing.

"There are several U.S. federal customers in sight" for this technology, Brammer said in an interview. Projects currently underway include communications network models for commercial satellite firms, he said. The modeling covers the network's basic physical layer-predicting signal-to-noise ratios and achievable data rates in various types of environments. The communications take place via the Internet, by wireless terrestrial channels and by space-to-ground channels.

"We worry about the fundamental physical communications channel and network issues [such as] computer-to-computer protocols and information security" that have to be included in the model, said Brammer.

Models are done for different purposes, he explained. "In some cases, you want to be able to predict the performance of a communications system or network that a customer wants to build." In other instances, customers want to predict the network's performance in various environmental conditions.

"In other cases, they already have a network, so we build a detailed model. In the event of an outage or emergency, we can help with tactical planning of how to restore services," Brammer said. "There are number of reasons for the simulation models."

High-fidelity models can predict failures, such as bottlenecks in transmission, bandwidth availability and whether there is enough computational power, he said. "They may be interested in trading performance vs. security." That means they must decide how to transmit information at different levels of security through a network. "you have to make sure that only the people who are authorized to get the information can get it quickly and easily," said Brammer. "That is easy to say, but very difficult to do in practice."

Modeling of computer attacks is done frequently, he said. "Some of these attacks have been well coordinated, involving thousands of computers. So we have to be able to model what might happen and what to do about it."

The challenge for the industry, he said, is to "understand how threats to the performance of communications systems are changing as a result of rapid changes in computing and communications technology." At the same time, he added, "a lot of the technology is so new and relatively immature, so there are security issues that you have to be careful about."

In information security, Brammer said, "we have a lot of learning to do."