A better cost-estimating tool: the key to not going over budget.

Department of Defense weapons sytems and program developments have received considerable unwanted attention, not because of innovative design issues, but rather, because rising costs concern those who control the money. In fact, cost overruns in some programs have pushed the price well beyond the original estimates.

[ILLUSTRATION OMITTED]

Historical Lessons of High Costs

This is not a new phenomenon to DoD. Weapons acquisition throughout U.S. history experienced numerous cost-estimate errors. Among the first contractors bedeviled by cost problems was Joshua Humphreys, the creator of the Navy's first six frigates. These ships were designed to be heavily armed and able to outgun any European ship of a similar class, yet fast enough to outrun any larger ship in a light breeze. The big frigates had special construction requirements that required live oak timber for critical components. According to Ian W. Toll, author of Six Frigates: The Epic History of the Founding of the U.S. Navy, several hundred live oak trees were needed for each ship.

There was a problem, though. Live oak grows best in the coastal plains of the southeastern United States. At the end of the 18th century and the beginning of the 19th century, harvesting the enormous live oak trees was a painstakingly slow and expensive proposition because the coastal plains had few roads and an abundance of disease-carrying mosquitoes. So expensive, in fact, that in 1794, the U.S. House of Representatives appointed a special committee to investigate how $7,000 could be spent on timber in a single month, especially when the estimated cost was much lower. Pointed queries were made, egos were wounded, and political posturing abounded, but the program edged on to a satisfactory completion.

Reasons for Cost-Estimating Problems

Looking at this example and numerous other occasions in DoD's history in which new weapons systems resulted in significant underestimations of costs, three factors appear to be associated with costing errors:

* A new technology or concept is introduced. In Joshua Humphreys' case, the innovation was the design of the ship and the materials required.

* Changes in design after the system is in production. These changes invariably result in unintended consequences and additional costs.

* The contractor routinely accepts that the technology is mature and there will be no design modifications after the system is produced. This results in an initial low-cost estimate that is very often unrealistic.

At the beginning of the 21st century, problems with cost-estimate accuracy, a history of cost growth, and high visibility have caused DoD program managers to seek more detailed cost estimates with frequent updates. However, manic attention to cost estimating also has its price. Program management and engineering personnel throughout the defense industry expend time, energy, and resources in developing, validating, and certifying cost estimates. The unplanned workload and reporting requirements are placing a strain on an overburdened logistics system. In many cases, programs simply have inadequate staffs to manage the contracts.

Not Enough Time

There is a growing need to develop cost estimates to support a variety of managerial, programmatic, and engineering requirements, and get it done fast. That's another cost-estimating problem. Routinely, the technical community receives a proposal to improve a weapons system's performance to meet emergent requirements. Once the proposal is briefed to the chain of command, the project engineer is asked, "How much will it cost?"

While searching for a quick response, the project engineer is asked another question: "How soon can you get me a rough order of magnitude [ROM] on the cost?"

The project engineer does a mental retrieval and concludes that a full bottoms-up engineering estimate is needed, but that will take too long--about three to four months. The project engineer knows it has to be faster, so he throws a number out. "I need a month to develop a ROM."

"Give me a ROM in two weeks if you really want any chance of funding this initiative," is the reply.

The project engineer walks away thinking, "How am I going to develop a sound, engineering-based cost estimate in two weeks?" Of course, the project engineer is correct. It will take several months, or even longer, to develop a realistic cost model of a complex system or major system upgrade, as well as the logistics tail associated with any change to a fielded system (spares, technical manuals, allowance parts lists, planned maintenance systems, integrated logistics support plans, and so on).

The related scenario is typical of situations encountered daily in defense industry engineering efforts. A tool is needed to enable a robust, engineering-based modeling and simulation of system-level technical characteristics, including the required performance parameters and associated costs. It must be an automated tool that helps streamline the existing laborious process of collecting component data and projecting technical cost for performance trades, and that assists in determining schedule considerations and technology maturity.

The Tools Are Out There

Many of the elements of the M&S system are readily available. There is an abundance of technical data available from component manufacturers and government research activities. The project engineer's vision can provide the required system parameters. With the data and expertise in place, all that is left is to establish the statistical relationships between the data, weight the relative importance of the data, and possibly develop a qualitative ranking schema. Thus, the resulting estimate will be based on historical and factual data, coupled with projections of new developments and price points from research activities and with the project engineer's vision for the desired architecture.

The algorithms and statistical modeling as well as the complex relational database storage and retrieval systems have been developed for other DoD applications over the past five years. It is feasible that a system-level M&S tool can be developed to actually model system-level performance and associated costs and schedule considerations. Taking advantage of the DoD tools available will allow the creation of an M&S cost-estimating system that can easily develop complex estimates in a timely manner.

[ILLUSTRATION OMITTED]

Once the base estimate is developed, the project engineer and program personnel can evaluate "what ifs" by varying the performance characteristics, thus testing the system's maximum performance potential with minimal cost and schedule factors. Performing the test in a digital date environment will create sufficient rigor in the estimates and will provide the ability to quickly and efficiently develop various scenarios with the data--including system-performance characteristics--resulting in a comprehensive system-level cost estimate.

Current Cost-Estimating Methodologies

Typically, cost estimates are developed by the selection of one or more estimating methods. The methods are usually determined by the amount and quality of available cost data. There are three common types of cost estimating:

* Parametric cost estimates use an equation to represent the cost-estimating relationships between one or more characteristics of a system to an element of its cost. An example of a parametric cost estimate may be an estimate based on the system's weight or the space it will occupy on board the ship.

* Analogy cost estimation compares the proposed system to an existing system that is similar in design or operation. For example, a proposed radar system may be evaluated against an existing similar radar system.

* Engineering cost estimates are based on detailed bottoms-up calculations and are the most time-consuming of the three techniques. Extensive amounts of detailed data and labor hours are required for this approach, and still the quality of the estimates is dependent on the credibility of the data available.

Modeling and Simulation-Based Solution

The ideal M&S-based cost-estimating system will use all three of the aforementioned cost-estimating methods, blending data stores of parametric data, existing systems costs (historical data), and detailed cost and technical data at the component level.

A preliminary systems design can be created based on the accumulated data and on a series of customer responses. The weighting of each of the models' cost-estimating methods will be adjusted based on the relative accuracy score of each specific data element. The result is a detailed, Composite cost estimate based on all available data at the time of the estimate. To validate the reliability of the model, actual historical cost data will be used to develop a cost estimate, and then historical actual cost data will be analyzed to refine the ability of the model to accurately predict future estimates. As actual data become available, the model will be updated automatically to ensure that the data weighing and statistical relationships are optimized.

Figure 1 is the proposed data construct for a radio frequency system components/modules cost-estimating M&S tool. The data construct shows the source and nature of the information to be collected and stored. There is a pathway of two-way communication between an M&S cost-estimating tool and all of the key data holders: academia, risk management analysis data stores, government research laboratories, statisticians, original equipment manufacturers and system developers, cost estimators, naval inventory control point, and commercial radio frequency manufacturers.

[FIGURE 1 OMITTED]

In Figure 1, a series of technical interchanges is conducted to ensure the development of the radio frequency system components/modules cost-estimating M&S tool, guided by expert knowledge and best industry practices. This knowledge is captured and leveraged via Lean Six Sigma and teaming arrangements.

A combination of models is used to create realistic cost estimates for weapons systems. Cost-estimating models used in the weapons system cost-estimating M&S tool will include a parametric model, an accounting model, a simulation model, and statistical simulations. The parametric model will contain a set of equations, each of which relates costs to parameters that describe the design, performance, operating characteristics, or operating environment of a weapons system. An accounting model will be included and will contain a set of equations used to combine elements of costs from simple relationships or directinputs. Some elements will be computed on the basis of unit cost and procurement quantity while others will be estimated using separate models or methodologies. These estimates are provided to arrive at an aggregate estimate of costs. A computer simulation model will determine the effect on costs of a system's characteristics, operational constraints, base concept, maintenance plan, and spare and support requirements. The simulation model will break costs into workable elements for which estimates are then developed using hardware parameters (such as reliability, maintainability, etc.). Statistical simulations will be used over time to generate probability density functions that describe the impact of system characteristics, operations, and maintenance concepts. Because of the large amount of data required, the use of such models is normally limited to the later program phases in which sufficient amounts of detailed data are available.

Putting the Tool to the Test

How effective is the M&S tool in developing complex cost estimates? Let's provide a proof-of-concept demonstration in which we'll develop ROM cost estimates on state-of-the-art radio frequency technologies to be inserted into existing and new radio frequency systems. The initial focus of this M&S effort will be on the radio frequency sub-systems and components of a radio frequency system.

To develop the M&S cost-estimating system, a systems engineering development analysis and initial development will be performed. A systems engineering approach to the cost-estimating procedure will ensure a disciplined methodology based on proven techniques. This process will require a number of studies, and it will include analyzing needs, operational requirements, functions, concept of operations, and performance. when the studies are complete and the lessons learned, conclusions and recommendations are evaluated. The development phase will begin.

If the initial development is successful, additional system modules will be developed using a spiral development, or a modified waterfall incremental build model, depicted in Figure 2. The actual steps in each block will be refined based on input and consultation with team members, academia staff, users, program office representatives, and expert data stores/management personnel.

[FIGURE 2 OMITTED]

Even though this is primarily a software development effort, the principles of systems engineering still apply. System engineering principles and methods would be applied to all aspects of the management and engineering development phases during the development of this project. As spiral development continues, the products will be built and improved upon to provide users with a more robust, accurate, and useful M&S cost-estimating tool. Enforcing the rigors of systems engineering, technical reviews will be conducted on the following: cost model assumption, data manipulation history, and the statistical cost curves used for cost elements and sub-elements. The technical reviews will be conducted by peers and subject matter experts.

The Changing Face of Cost-Estimating Methods

Both historically and today, DoD is facing a persistent problem of inaccurate cost estimates based on outmoded and inadequate methods. However, data and techniques exist that can improve the process and accuracy of weapons systems cost estimation. The storage of all cost-estimate models will allow timely updates and refinement of the overall model. The alternative is to continue with the current cost-estimating methods, and we can expect many of the same results that plagued Joshua Humphreys and his six frigates at the beginning of our Navy's history: Cost overruns, missed schedules, pointed inquiries, wounded egos, and political posturing are the inevitable companions of poor cost estimates.

The authors welcome comments and questions and can be contacted at james.m.young@navy. mil and [e-mail].

Young is an integrated logistics manager at Naval Surface Warfare Center Crane Division, Crane, Ind. For the past three years, he has been supporting the Office of Naval Research as a life cycle cost lead for innovative naval prototypes. Markley is a senior defense analyst with Lockheed Martin Training Solutions. Inc.. at the Naval Surface Warfare Center Crane Division, Crane, Ind.