SE and EVM support for performance-based awards.

On March 29, 2006, DoD issued a memorandum directing that award fee contracts be structured to focus government and contractor efforts on meeting or exceeding cost, schedule, and performance requirements; and that award fees be linked to achieving desired program outcomes. This was buttressed by the DoD Appropriations Act of 2007, which prevents payment of award fees for performance that does not meet the requirements of the contract (Sec. 9016). Systems Engineering (SE) standards and Earned Value Management (EVM) provide a framework for linking award fees to desired program outcomes. This article provides practical advice for defining the technical performance requirements and desired program outcomes in SE terms. It updates information that was published in "Integrating SE with EVM," Defense AT & L, May-June 2004.

[ILLUSTRATION OMITTED]

GAO Findings and Resultant DoD Policy

The DoD policy and guidance follows Government Accountability Office recommendations. GAO studied failures in procurement of weapons systems and Information Technology systems. Recent reports (GAO Reports 06-66, 06-391, 06-110) disclose recurring weaknesses in procurement management and provide recommendations for achieving desired outcomes. Some GAO findings and recommendations are summarized below.

* Finding: Contractors are not held accountable for achieving desired outcomes, including cost goals, schedule goals, and desired capabilities.

* Finding: Programs do not capture, early on, the requisite knowledge needed to effectively manage program risks.

* Finding: DoD needs to change its requirements and budgeting processes to get desired outcomes from the acquisition process.

* Recommendation: Capture knowledge about completion of subsystem and system design reviews.

* Recommendation: Agree that drawings are complete.

* Recommendation: Demonstrate with prototype that design meets requirements.

The resultant DoD policy and guidance directs the following: (1) Award fees must be linked to desired interim outcomes, discrete events, and milestones. (Examples of interim milestones are timely completion of Preliminary Design Review (PDR) and Critical Design Review (CDR).) (2) Progress toward interim milestones must be assessed. (3) Award fee provisions must clearly explain how a contractor's performance will be evaluated.

If a program manager specifies contractual requirements for the conduct of a complete, integrated SE effort, and integrates SE with EVM, award fees can be linked to interim outcomes, discrete events, and milestones. It is possible to ensure that the reported earned value truly integrates technical performance with schedule and cost performance. When SE is integrated with EVM, earned value and its derived measures--such as the cost-performance index--can provide a valid, objective basis for linking award fees to desired outcomes.

DoD Guides

DoD guidance for integrating SE with EVM is included in the Defense Acquisition Guidebook (DAG); the SE Plan Preparation Guide; the Work Breakdown Structure Handbook, MIL-HDBK-881 A; and the Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide. The guides provide discretionary best business practices, as summarized in the figure on this page.

Integrated Baseline Review (IBR)

An important milestone for award fees should occur shortly after authority to proceed. Per the Federal Acquisition Regulation, the IBR is a joint assessment of the ability of the project's technical plan to achieve the objectives of the scope of work and the degree to which the management process provides effective and integrated technical/schedule/cost planning and baseline control. The IBR may also be use to verify that contractual requirements for the conduct of a complete, integrated SE effort have been incorporated into the baseline. These objectives should be criteria for award fees.

Standards and Best Practices

The following SE standards were adopted by DoD and are cited in the DAG: Electronic Industries Alliance Processes for Engineering a System (EIA 632) and the Institute of Electrical and Electronics Engineers Standard for Application and Management of the SE Process (IEEE 1220). They provide guidelines and best practices for using product metrics, including technical performance measures (TPMs), and for defining completion criteria for PDRs and CDRs.

TPMs and Product Metrics

The guidelines and best practices for product metrics from EIA 632 are to identify and track TPMs to determine the success of the system; project the evolution of the parameter as a function of time toward the desired value at the completion of development; and to identify product metrics and their expected values that will affect the quality of the product and provide information toward satisfying acquirer and other stakeholder requirements, as well as derived requirements.

IEEE 1220 includes similar guidance on TPMs and product metrics. It also discusses the need for progress measurements of design maturity.

Completion Criteria for Technical Reviews

IEEE 1220 describes tasks that should occur during all technical reviews. The outcome of these tasks can be used to determine award fees. The tasks are to assure that all master schedule success criteria have been met; assess development maturity to date; assess the product's ability to satisfy requirements; and assure traceability of requirements and validity of decisions. IEEE 1220 provides specific guidance and exit criteria for PDRs and CDRs, as follows.

The PDR Subsystem review assures that subsystem definition is sufficiently mature to meet SE master schedule criteria; component allocations and preliminary component specifications provide a sound subsystem concept; subsystem risks have been mitigated; trade-study data substantiate that subsystem requirements are achievable; and decisions made in arriving at the subsystem configuration definition are well-supported by analysis and technical data.

The PDR System review takes place after completion of subsystem reviews. Its purpose is to determine whether the total system approach to detailed design satisfies the system baseline; unacceptable risks are mitigated; issues for all subsystems, products, and life-cycle processes are resolved; and accomplishments and plans warrant continued development effort.

[ILLUSTRATION OMITTED]

The CDR Component review ensures that each detailed component definition is sufficiently mature to meet measure-of-effectiveness/measure-of-performance criteria; component specifications provide a sound component concept; component and related life-cycle process risks have been mitigated to a level appropriate to support Fabrication, Assembly, Integration and Test (FAIT); trade-study data substantiate that detailed component requirements are achievable; and decisions made in arriving at the detailed component definition configuration are well-supported by analysis and technical data.

The CDR Subsystem review follows the component reviews and determines whether the subsystem detailed design satisfies the design-to baseline; risks are mitigated and remaining risks are acceptable; issues for all components, assemblies, and life-cycle processes are resolved; and accomplishments and plans warrant continuation with FAIT.

The CDR System review takes place after completion of subsystem detailed design reviews to determine whether the detailed design of the system satisfies the system baseline; unacceptable risks are mitigated; issues for all subsystems, products, and life-cycle processes are resolved; accomplishments and plans satisfy criteria for continuation of the technical effort; and the system is ready to continue into FAIT by having resolved outstanding product or life-cycle process issues.

Technical Performance-Based EV

The SE standards have common elements for basing earned value on technical performance; the use of product metrics, including TPMs; measurement of quality and design maturity; and definition of exit criteria for technical reviews.

An important control for ensuring integration of a project's technical performance objectives is to use these elements as exit criteria for work packages and for interim progress measurement. For example, the completion criteria of a work package should include both the enabling work products, such as drawings or software code, and meeting the requirements, such as weight limits or the allocated functional requirements. When earned value is based on technical performance, it will be a valid, reliable indicator of program status.

Earned value can also be a valid basis for award fee determination if it is tied to technical performance, not just to work accomplished. The 2004 Defense AT & L article mentioned earlier cautioned that EVM data will be reliable and accurate only if the right base measures of technical performance are selected and if progress is objectively assessed. If you are measuring the wrong things or not measuring the right way, then EVM may be more costly to administer and may provide less management value. The GAO had similar findings regarding EVM and technical performance goals. GAO Report 06-250 found that EVM can have an impact on acquisition success if properly implemented; however, if not implemented effectively, decisions may be based on inaccurate and potentially misleading EVM information.

Performance-based earned value will meet the Office of Management and Budget Circular No. A-11 requirement for a performance-based acquisition management system based on EVMS, for capital investments that measure progress towards milestones in terms of capability of the investment to meet specified requirements and quality.

Contractual Performance-Based Progress and Incentives

DoD customers should use performance-based acquisition management by including requirements and award fee incentives for performance-based management and reporting in their contracts, beginning with the solicitation. Then the program manager can link award fees to achieving desired program outcomes. Earned value will provide insight that is based on technical performance if the contractor is required to link discrete work packages to milestones for key technical and management deliverables. A sample of those deliverables follows:

* Success criteria for major technical reviews

* TPM planned values and measurement milestones

* Master schedule that identifies all systems engineering products, such as the technical baselines and requirements traceability matrices; identifies TPM planned value milestones; and is linked to the identified success criteria

* Product metrics reports.

The Air Force Space and Missile Systems Center, Air Force Space Command, published and uses a comprehensive Technical Operating Report (TOR) that specifies contractual requirements for the conduct of a complete, integrated SE effort. The requirements are defined in terms of the required SE products and the required attributes of those products. For example, it states that "the Contractor SHALL monitor the progress against all planning" and prepare documented assessments that include TPMs and "metrics and selected technical parameters for tracking that are critical indicators of technical progress and achievement." The TOR is used to prepare the requests for proposal and for evaluating the contractor's SE products once on contract.

The TOR is an excellent document for defining and monitoring the contractor's SE efforts. I recommend that the contractually required TPMs and metrics be used for award fee determination. The TOR is available at <www.PBEV.com> within PBEV Resources. If a program manager specifies contractual requirements for the conduct of a complete, integrated SE effort, then award fees may be used to focus contractor efforts on meeting or exceeding cost, schedule, and performance requirements.

The author welcomes comments and questions. Contact him at [e-mail].

Solomon oversees EVM on Northrop Grumman Corporation programs. He is an author of the EVMS Standard and the book, Performance-Based Earned Value[R]. He is a recipient of the Department of Defense David Packard Award for Excellence in Acquisition.

Policy or Guide                           Policy  DAG  SEP  WBS  IMP/IMS

Event-driven timing of technical reviews  X       X    X    X    X
Success criteria of technical reviews     X       X    X    X    X
Assess technical maturity in technical            X    X    X
reviews
Use TPMs to compare actual vs. planned            X    X         X
technical development and design
maturity
Use TPMs to report degree to which                X    X
system requirements are met in terms of
performance, cost and schedule
Integrate SEP with IMP, IMS, TPMs, EVM            X    X         X
Integrate WBS with requirements                             X    X
specification, statement of work, IMP,
IMS, and EVMS.

Guidance for Integrating SE with EVM