Letter to the editor.

In your January-April 2004 edition of the Defense Acquisition Review Journal article entitled, "Surveying Cost Growth," the authors indicate that one way to reduce unexpected cost growth is to develop better estimates. They go on to quantify some of the reasons for this growth. According to a 1993 RAND study of Acquisition Category I (ACAT I) programs, by the time these programs complete the production and fielding phases, they will experience approximately a 20 percent cost growth from the initial estimate.

I would like to point out that many studies have been done in the past few years that reflect similar results, and in comparison to other complex public and private sector industries, the Department of Defense (DoD) overruns are relatively small. In a Technical Report (TR 1-00) written for the Defense Acquisition University (DAU) in October 2000, entitled, "Acquisition Trend Metrics in the Department of Defense," it was shown that for engineering and manufacturing development (EMD) programs in the DoD between 1980-1996, the average cost overrun was 40 percent. But, if the nine programs that had over 100 percent overruns were removed (considered outliers), then the average overrun was reduced to 19 percent. That would appear to indicate that either a program's cost overrun as a percentage remains fairly consistent from EMD through Production/Fielding (RAND study) or that the bulk of the program's overruns occur early-on during development and gets no worse during production.

That is not to say that there is not room for improvement because we are continuously trying to improve our cost-estimating methodologies and databases. Unfortunately, the development of new and useful Cost Estimating Relationships (CERs), the positive use of good risk analysis, and other techniques are often lacking for different reasons. Sometimes it is totally dependent on the experience of the analyst(s) who are trying to develop the estimate.

Cost estimates, by their own nature, are not precise, discrete decision-making tools. Particularly in the early stage of a program pre-System Development and Demonstration (pre-SDD or SDD) phase, there is usually very little information of a technical or cost nature to use with any confidence level. Thus, there is risk inherent in the process. Cost Analysts can and often do quantify risks in their programs. Army ACAT I level programs are traditionally funded to the 50 percent confidence interval. But even this is no guarantee that a program is adequately funded. This is equal to the flip of a coin. A program therefore starts on a playing field that is expected to have a 50/50 chance of not having a cost overrun. And as most acquisition managers can attest, the likelihood of the program having some significant changes to it during the initial years of SDD is great.

The design of the system is immature, and the details of how reliable the hardware will eventually become has not been determined. What specific kinds of testing the Program Manager (PM) and other organizations will conduct to determine the system availability, maintainability, and ability to meet technical requirements are all areas of uncertainty. At the same time, the Program Office Estimates (POEs) are being finalized for management review and approvals. Unfortunately, this becomes the baseline funding for the program and gets hardened in cement shortly afterward. A PM's ability to influence and significantly change a funding profile for anything other than 4-5 years away is quickly reduced to time zero.

I would also point out that in TR 1-00 the authors indicate, "cost growth is usually a consequence of some other occurrence and not a lead event" (Swank, Alfieri, Gailey, & Reig, 2000, p. 2). In other words, cost status is not the predictor of a program's problems but rather the result of the program's problems. Those problems are usually either schedule- or performance-related activities. A design problem that cannot be solved or a prototype of a new circuit card assembly that keeps cracking during a stress test can be the cause of a serious technical problem. On a similar note, schedule status is the best early indicator of a program in trouble, "... a slip in early test milestones is the best indicator of a program heading into trouble" (Swank, Alfieri, Gailey, & Reig, 2000, p. 4).

What we can say about trend analysis from an Earned Value Management perspective is that when a contract is more than 15 percent complete, it is a widely accepted principle that the percent overrun at contract completion will be greater than the percent overrun incurred to date.

In addition, the software complexity of today's systems cannot be overstated. Many of the studies were taken from Selected Acquisition Reports (SARs). It should be noted that SARs do not ask for or contain information specific to a program's software development. As a result, a major cost driver to technology in modern systems cannot be clearly delineated or identified to any degree. Today's weapons systems are becoming more software intensive and complexities, due to software estimating at an early stage of development, are difficult and often underestimated.

As for the underestimating of costs on programs, we should also consider that the analyst is often given constraints while developing estimates. For example, by the time a formal estimate is required for a Milestone B decision, certain near-term funding had already been approved a few years earlier. These budget wedges are usually created from engineering best guesses and ROMs created to serve as a beginning to the financial programming process, much earlier than the technical requirements, program schedule, and systems definition are clearly defined. As a result, the analyst is often forced to estimate to budget in the near term, even as requirements dictate that more dollars are needed to be programmed for the work expected. Thus, even if risk dollars are needed, the funding may not support it, and the program may not be properly funded for unknowns likely to occur. This then appears as a cost estimate that is not aligned with optimistically projected programmatics. The result of all of these constraints and what-if drills may be the cause of the estimate being low in certain fiscal years and high in out-years, when funding changes can be impacted.

While more realistic (i.e., accurate) cost estimates are certainly a commendable goal, the issue of affordability and funding programmatics tend to negate each other in the final analysis.

REFERENCES

Drezner, J. A., Jarvaise, J. M., Hess, R. W., Hough, P. G., & Norton, D. (1993). An analysis of weapon system cost growth. RAND Study MR-291-AF. Santa Monica, CA: RAND Corporation.

Greiner, M., Sipple, V., & White, E. (2004, January-April). Surveying cost growth. Defense Acquisition Review Journal, 11(1), pp. 79-90.

Swank, W. J., Alfieri, P. A., Gailey, C. K., & Reig, R. W. (2000, October). Acquisition trend metrics in the Department of Defense (TR 1-00). Fort Belvoir, VA: Defense Acquisition University/Defense Systems Management College.