Integrating systems engineering with earned value management.

Program managers (PMs) expect their supplier's earned value management system (EVMS) to accurately report the program's integrated cost, schedule, and technical performance. However, EVM data will be reliable and accurate only if the right base measures of technical performance are selected and if progress is objectively assessed. If you are measuring the wrong things or not measuring the right way, then EVM may be more costly to administer and may provide less management value.

[ILLUSTRATION OMITTED]

During my experience monitoring EVM on many programs. I often observed programs that were behind schedule in terms of validating requirements, completing the preliminary design, meeting weight targets, or delivering software releases that met the requirements baseline. Yet 100 percent of earned value was taken and reported, in compliance with the industry standard for EVMS, because the EV completion criteria were not based on technical performance or were not defined clearly and unambiguously. Furthermore, during technical reviews, some of these adverse conditions were not described as problems or issues. They were classified as risks towards achieving subsequent objectives.

EVM can be more effective as a program management tool if it is integrated with technical performance and if the EVM processes are augmented with a rigorous systems engineering process. The recommendations that follow are based on lessons learned from major programs and on observing the processes of major contractors and subcontractors. Guidance is provided for PMs to ensure that reported EV is a valid indicator of technical performance. Pre-contract and post-contract actions are recommended to implement performance-based earned value that is quantitatively linked with:

* Technical performance measurement (TPM)

* Progress against requirements

* Development maturity

* Exit criteria of life cycle phases

* Significant work packages and work products.

Guidance for getting more value out of earned value is consistent with the Department of Defense (DoD) Risk Management Guide (Guide), the Interim Defense Acquisition Guidebook (IDAG), and with industry standards that have been adopted by the DoD:

* Processes for Engineering a System (EIA 632)

* Standard for Application and Management of the Systems Engineering Process (IEEE 1220)

* EVMS (ANSI/EIA-748-A-1998).

Additional guidance is consistent with the Capability Maturity Model[R]-Integration (CMMISM).

Better integration of systems engineering, risk management, and EVM will benefit the PMs of both the acquisition and supplier organizations.

EVM Limitations

With regard to a PM's needs, there are several limitations of EVMS that can be overcome by integrating EVM with robust systems engineering. First, EVM is perceived to be a risk management tool. However, EVMS was not designed to manage risk and does not even mention the subject.

Unfavorable cost or schedule variances result from past events. They are already problems or issues. A cost overrun indicates that, with 100 percent probability, subsequent cost objectives will not be achieved unless the plan for remaining work is revised.

Second, earned value is a derived measure. Consequently, its effectiveness to integrate technical and cost performance depends on its base measures and on the capabilities of the systems engineering processes that are employed on a program.

[FIGURE 1 OMITTED]

Third, EVMS does not require precise, quantifiable measures. It states that objective earned value methods are preferred but it also states that management assessment (subjective) may be used to determine the percentage of work completed.

Finally, EVMS states that EV is a measurement of the quantity, not quality, of work accomplished. A PM should ensure that EV also measures the quality and technical maturity of technical work products instead of just the quantity of work. Robust systems engineering processes should provide TPM and exit criteria for assessing technical maturity that are quantitatively linked to EV.

The following guidance will help a PM overcome EVM's limitations.

Risk Management Guide and TPM

Per the Guide, risk management is concerned with future events whose outcome is unknown and with how to deal with these uncertainties. That guidance is in contrast to risk-handling actions that should be reflected in integrated program planning, scheduling, and work packages. In other words, risk handling actions become part of the EV performance measurement baseline (PMB).

In my opinion, the Guide's statement that "periodic EV data can provide indications of risk" is misleading. As discussed above, by the time a cost overrun is reported, the unfavorable event has occurred and there is a problem or issue, not simply a risk.

The same premise--that deviations from a plan are issues, not risks--should apply to TPM. Per the Guide:

* Technical ... parameter values to be achieved ... are forecast in the form of planned performance profiles.

* Achieved values for these parameters are compared with the expected values.

* Events, tasks, and schedule resulting from the integrated planning are linked with ... techniques, such as TPM.

* Linkage provides a significant monitoring tool, giving specific insights into the relationships among cost. schedule, and performance risks.

An example of a TPM planned performance profile that also shows achieved values and a tolerance band is shown in Figure 1.

However, some PMs classify TPM as a risk management technique and do not integrate the planned performance profile into the schedules and work packages. Later, if achieved values for these parameters fall short of the expected values, neither the schedules nor the earned value show a behind-schedule condition.

Mike Ferraro describes DCMA research and pilot tests for integrating TPM and EVM ("TPM, a PM's Barometer," PM, November-December 2002). The earliest research, published in 1995, found that there was not clear linkage between technical parameters and work packages. Ferraro concluded that this continues to be an issue.

So how can a PM obtain contractual commitment to integrate TPM and EVM? Fortunately, there are two industry standards that provide specific guidance for TPM that are consistent with the Guide: IEEE 1220 and EIA 632. Both standards provide guidance for TPM planning and measurement (Figure 2) and for integrating TPMs with EVM. The DoD has adopted both standards.

A PM may require compliance with the TPM components of either of these standards in the solicitation. Another approach is to provide financial incentives for contractor compliance. After contract award, the PM may use the integrated baseline review (IBR) to verify that the integrated planning includes TPMs and that the EVM is quantitatively linked to achieved values in appropriate work packages. If the PM uses simulation-based acquisition and modeling & simulation as discussed in IDAG, then the achieved values should be credible. Finally, the PM should address TPM achievement and reporting during technical assessment reviews.

Other Systems Engineering Best Practices

IEEE 1220 and EIA 632 provide additional guidance for systems engineering process improvement regarding progress, planning, and measurement. It may be used to select performance-based earned value measures. A PM may choose to mandate compliance with pertinent components of the standards in the solicitation or to provide other incentives for compliance.

Progress Against Requirements

Master schedules and PMBs often reflect the tasks that were proposed, estimated, and negotiated. However, tasks that formed a basis of estimate for negotiation are not necessarily those that should be planned and tracked during program execution. The PM should select base measures of progress for EV that indicate objective progress towards development, implementation, and testing of the requirements.

The Guide discusses product-related metrics that include requirements traceability and requirements stability. Progress against requirements, including the percentage of requirements that are traced upwards and downwards and those that are validated, would be a highly effective base measure of earned value. It is especially important to validate the requirements baseline early in development and prior to the start of design by the prime and subcontractors.

The industry standards' guidance for assessing progress against requirements is shown in Figure 3 (page 46).

Design Maturity

The Guide discusses design maturity as a product-related metric and provides examples of design maturity measures. Adherence to the standards will support the requirement in DoD Instruction (DoDI) 5000.2 for a design readiness review during system development and demonstration. The design readiness review assesses design maturity as evidenced by such measures as:

* Number of subsystem and system design reviews successfully completed

* Percentage of drawings completed

* Planned corrective actions to hardware/software deficiencies

* Adequate development testing.

Objective assessment of a system's design maturity, in compliance with the standards, would also be a sound basis for earned value.

Exit Criteria

The standards discuss the importance of holding technical reviews at the end of a stage of development or a life-cycle phase to assure that all exit criteria have been met. IEEE 1220 is especially helpful by providing exit criteria for a preliminary design review (PDR) and a detailed design review. Some of the exit criteria for a PDR are:

* Prior completion of subsystem reviews

* Determination whether total system approach to detailed design satisfies the system baseline

* Mitigation of unacceptable risks

* Resolution of issues for all subsystems, products, and life cycle processes

* Definition of exit criteria in a systems engineering management plan or other technical plan.

A PM should review these plans with the supplier and reach agreement on the validity and sufficiency of the exit criteria during the IBR. It is also recommended that the work packages that measure progress against requirements and development maturity be reviewed to understand the time-phased plan for meeting the exit requirements, the related EV techniques, and the base measures.

[ILLUSTRATION OMITTED]

Systems Engineering Work Products

The systems engineering process generates significant work products that should be included in integrated planning and measured with earned value.

The process products of IEEE 1220 are:

* Requirements baseline

* Validated requirements baseline

* Functional architecture

* Verified functional architecture

* Physical architecture

* Verified physical architecture.

The process products of EIA 632 are:

* System technical requirements

* Logical solution representations

* Physical solution representations

* Specified requirements

* Validated system technical requirements

* Validated logical solution representation

* Verified design solution.

Depending on the selected standard, these work products should be included in the master schedule and in work packages. Additional recommendations for work products are provided below in a discussion of the CMMI.

Bad Rap for Level of Effort (LOE)

Many PMs expect that the percentage of LOE budget should not exceed a certain level. I believe that setting an arbitrary maximum threshold for LOE can increase contract costs and cause management to waste time by focusing on the wrong things. It costs money to measure processes and progress. But as Navy Rear Adm. Dave Antanitus wrote in PM, "Be careful here--just because you can measure something does not mean it is a useful metric!" ("The Business of Metrics," March-April 2003).

Many tasks that are measurable are not indicators of technical performance. Examples are technical assessment meetings and recurring reports, such as cost performance reports (CPR). If a CPR is delivered late, there is no schedule impact on a subsequent activity and no impact on final costs. So why incur the costs to measure CPRs discretely or to analyze schedule variances?

The same is true for technical assessment reviews, such as technical interchange meetings (TIMs), PDRs, and final design reviews. Per IEEE 1220 and EAI 632, a purpose of the reviews is to assess progress and development maturity. However, it is common practice to base earned value on completion of the milestone event (TIM or PDR was held) instead of on the quantified assessment of progress and maturity. For a PDR, if earned value were based on the event instead of the assessment and if the preliminary design did not meet the exit criteria, then earned value would mask a behind-schedule condition. Likewise, the master schedule would be misleading if the PDR event showed completion despite a shortfall in technical performance.

It would be cheaper to designate non-technical tasks as LOE, to manage LOE cost performance, and to apply more management attention to technical performance. Both EIA 632 and IEEE 1220 focus on technical progress. The budget for non-technical tasks, such as preparing for and conducting a PDR, could be planned as LOE even if the LOE percentage exceeded arbitrary limits. The EVMS standard discusses that LOE is supportive work that is impracticable to measure. Non-technical work may fit this definition.

A PM should be careful when analyzing summary earned value information. A summary of only the discrete tasks that measure technical performance should be prepared. The performance-based earned value will show schedule and cost variances that are not distorted by LOE content. Also, the related cost performance index will be a truer indicator of future costs. LOE should be summarized and analyzed separately.

Additional Resources

The industry standards provide information as to what to do, and they provide a basis for acquisition management. Process models like CMMI provide information for implementing processes. The CMMI provides a framework for process improvement towards integrating systems engineering and EVM.

The Carnegie Mellon Software Engineering Institute's publication Using CMMI to Improve EVM (<www.sei.cmu.edu/>) provides information on the following processes and topics:

* Requirements development

* Requirements management

* Measurement and analysis

* Process and product quality assurance

* Risk management

* Typical work products

* Performance-based earned value.

Guidance for requirements-based planning is provided in "Practical Software Measurement, Performance-Based Earned Value" (CrossTalk: The Journal of Defense Software Engineering, Sept. 2001, <www.stsc.hill.af.mil/crosstalk>).

A contractor may be compliant with EVMS but fail to truly integrate measurement of cost, schedule, and technical performance. A PM should ensure that integrated plans, schedules, and the earned value PMB are linked with the contract requirements, TPMs, and unambiguous exit criteria. By requiring or encouraging suppliers to adhere to industry standards for systems engineering or engineering processes, EVM will provide more reliable information.

FIGURE 2. TPM Planning and Measurement

IEEE 1220: 6.8.1.5                   EIA-632: Glossary

Performance-based progress
measurement
TPMs are key to progressively        Predict future value of key
assess technical progress            technical parameters of the end
                                     system based on current
                                     assessments
* Track relative to time with dates  * Planned Value profile is time-
  established as to when:              phased achievement projected
  -- Progress will be checked        * Achievement to date
  -- Full conformance will be met    * Technical Milestone where
* Use to assess conformance to         TPM evaluation is reported
  requirements

FIGURE 3. Progress Against Requirements

IEEE 1220                        EIA 632

6.8.1.5 Performance-based        4.2.1 Planning process,
progress measurement             Req. 10: Progress against
6.8.6 Track Product ... Metrics  requirements

6.8.1.5 d) Assess:               *  Assess Progress ... comparing
* Development maturity to date      currently defined system
* Product's ability to satisfy      definition against requirements
  requirements                   a) Identify product metrics and
6.8.6 Product metrics ... at        expected values:
pre-established control points
  enable:                        *  Quality of product
* Overall system quality
  evaluation                     *  Progress towards satisfying
* Comparison to planned goals       requirements
  and targets                    d) Compare results against
                                    requirements

Editor's note: The author welcomes comments and questions and can be reached at [e-mail].

Solomon manages EVMS within the Northrop Grumman Corp., and is a visiting scientist at the Software Engineering Institute. He won the DoD David Packard Award with the team that wrote EVMS. He holds a bachelor's degree and a master's in business administration from Dartmouth College and is a project management professional (PMP).