Improving the integrity of the CMMI Product Suite.

Over the past few years, it has become apparent to the government sponsors of the Capability Maturity Model Integration that acquirers had unrealistic expectations of what a supplier's CMMI maturity level claims could provide to an acquisition program. The Department of Defense recognized that many DoD acquisition programs are including requests for CMMI maturity levels in requests for proposal (RFPs) in spite of the fact that DoD has not promulgated policy requiring adherence to any CMMI maturity level rating. DoD does not place significant emphasis on capability level or maturity level ratings, but rather promotes CMMI as a tool for internal process improvement. This lack of emphasis on ratings is prudent in the light of findings that not all suppliers are exhibiting behavior consistent with their attained CMMI maturity level rating. Additionally, issues have arisen regarding appraisal integrity and misrepresentation of the benefits of a "level."

DoD and industry have initiated various efforts to better understand the issues contributing to the difference in expectations versus DoD's observations. First, the National Defense Industrial Association (NDIA) Workshop and Summit on CMMI Use in Acquisition, held in September 2005, identified issues and offered a set of preliminary recommendations. The initial set of issues that DoD and industry determined to require further attention were:

[ILLUSTRATION OMITTED]

* Program office understanding of the need for and benefit of mature and capable processes

* Lack of training and guidance for acquiring organizations on CMMI usage

* Limited content and usefulness of the appraisal disclosure statements (ADSs) submitted by suppliers

* Benefits and drawbacks of specifying or requiring minimum CMMI maturity level ratings in RFPs

* Organizational approach to CMMI implementation and appraisals.

In the spring of 2006, a government review and assessment team validated these issues, refined and augmented them, then recommended actions to the CMMI Steering Group, which is composed of representatives from DoD, industry, and sponsors. Finally, the Defense Contract Management Agency conducted a data call that confirmed their initial assessment about inconsistencies between appraised ratings claimed and observed program performance.

These efforts resulted in a consolidated list of issues that the sponsors believed needed to be effectively addressed:

* Observations that suppliers execute at lower maturity levels than they have achieved in a formal appraisal

* Once achieved by an organization, a CMMI maturity level rating existed for life

* Organizations do not necessarily incorporate CMMI-appraised processes on new projects

* CMMI-driven processes and practices are not consistently applied at the project level after contract award

* Appraisal sampling procedures did not ensure adequate coverage of the organizational unit granted the appraisal rating

* Appraiser quality and training raised issues with integrity of maturity ratings

* Lack of consistent understanding and application of high-maturity practices

* Lack of definition within the CMMI Product Suite on what constitutes high maturity

* Content of ADSs lacks specificity and is not useful to program offices

* Inadequate training and education for acquirers.

CMMI Version 1.2 Release

The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, and the NDIA Systems Engineering Division--the co-sponsors of the CMMI--directed the CMMI Steering Group to take the necessary actions to improve the integrity of appraisal results and address other shortcomings in the CMMI Product Suite. Implementation teams under the stewardship of the Steering Group developed the necessary changes to the model, the appraisal method, and the training with this goal in mind. As a result, the CMMI Product Suite has been modified to address each of the identified issues.

"Level for Life" Eliminated

Many organizations have attained a CMMI maturity rating over the years. Often these ratings are prominently displayed in a firm's marketing material and highlighted in proposals. Since there was no definitive rule regarding when appraisal results became invalid, it was left to interpretation when appraisal results were considered stale. Version 1.2 established that appraisal results would remain valid for a maximum of three years. Organizations will have to re-appraise using the current CMMI model and appraisal method in order to make any maturity or capability level claims beyond this.

CMMI Version 1.1 Retired

The entire CMMI Version 1.1 Product Suite will be retired effective Aug. 31, 2007. Thus, all appraisals after that date must use the Version 1.2 model and appraisal method. The full set of CMMI Version 1.2 policies is available at <www.sei.cmu.edu/cmmi/appraisals/cmmiv11-sunsetappraisal-policies.html >.

<www.sei.cmu.edu/cmmi/appraisals/cmmiv 11-sunset-appraisal-policies.html>.

Organizational Processes Applied at Program Start-up 

Organizations at maturity Level 3 or higher are expected to employ their organizational processes on new programs. There is evidence that some organizations either do not employ them or delay applying them on new programs. Material added to the model in Version 1.2 now requires organizations to deploy their set of standard processes at project start-up and deploy changes to those processes as appropriate throughout the life of the project.

Appraisal Disclosure Statement Improved

The ADS was enhanced by the addition of specific information on the sampling approach, percentage of projects sampled, and the organizational scope. The updated Version 1.2 ADS requires provision of detailed information on what was appraised in an organization. Previously, it was often not clear which organizational unit actually had attained the maturity rating in a large corporation. The ADS will now list specific projects, organizational units, and domain information. This provides insight to an acquirer who is interested in the capability of a particular organization for a specific project, unit, or domain. Additionally, these changes in the ADS are intended to improve program selection integrity by placing more responsibility on lead appraisers to ensure a representative organizational sample. New, required disclosure information regarding applicability to the organizational unit clarifies this issue.

Verification of Level 4 and Level 5 Processes

To achieve a high maturity rating, an organization must quantitatively manage (maturity Level 4) or optimize (maturity Level 5) select processes and subprocesses. Review of CMMI appraisals revealed a lack of consistency in what constituted Levels 4 and 5. Some organizations would select one subprocess, while others would attempt to quantitatively manage and optimize all of their processes. Neither behavior exhibits the intent of the model, which is to apply high maturity practices on select processes or subprocesses that are important to an organization's business goals and objectives. Some processes may not warrant high maturity application. Further, if the chosen processes or subprocesses are not those of value to the organization, domain, or program, the supplier's high maturity activity may not add benefit. With Version 1.2 of the ADS, lead appraisers now have the added responsibility to verify that the chosen processes relate to the organization's business objectives. Additionally, the Software Engineering Institute (SEI), designated by the sponsors as the CMMI Product Suite steward, has undertaken a significant effort to set qualifications for high maturity appraisers, update high maturity training with certification, and deploy these standards across the pool of high maturity appraisers.

Appraisal Results Reporting Changes

Acquirers should have insight into all the process areas that make up an organization's capability or maturity level ratings. A capability level profile provides the capability levels of the individual CMMI process areas. Examination of the capability level profile provides the ability to determine whether an organization is mature in processes that are critical or relevant to an acquirer's program. A single maturity level rating does not provide this insight. In this regard, the capability profile level is more meaningful and is endorsed by the DoD as a better metric to understand process capability than relying on a single maturity level rating.

Quality Audit Review

The SEI, the CMMI steward, must review and accept results before appraised organizations will be allowed to publicly announce the appraisal results or use the information in a formal proposal to the government or other acquirer. This allows the SEI to identify and correct any inconsistencies in the appraisal process before the results are declared official.

DoD Contractor ADSs Posted

Firms that contract with the DoD and seek to have a recent CMMI appraisal considered are now required to post their ADS on the SEI's Published Appraisal Report Site (PARS) at <http://sas.sei.cmu.edu/pars/pars.aspx> for government acquirer review. This reporting was formerly optional, at the behest of the contractor. Acquirers are now encouraged to check the PARS site to validate a supplier's maturity level claim, since only validated results are posted on PARS. In addition, if the appraisal was conducted after Nov. 1, 2006, acquirers will be able to gain the appraisal specifics available in Version 1.2 of the ADS, since the use of the new ADS is mandated as of that date.

Lead Appraisers to be Independent

Lead Appraisers from the same organization as the one sponsoring the formal appraisal can no longer grant a capability or maturity level rating to their organization. It was determined by the CMMI Steering Group and sponsors that this independence is necessary to ensure integrity of appraisal results. The requirement applies only to the appraisal lead and not to the members of the appraisal team.

Guidebook for Acquirers Published

Understanding and Leveraging a Supplier's CMMI Efforts: A Guidebook for Acquirers was a major effort to help acquirers benefit from a supplier's use of CMMI while avoiding the pitfalls associated with unrealistic expectations. The Guidebook describes CMMI fundamentals and much of the information summarized in this article to help acquirers effectively use information obtained from a supplier's CMMI effort. It includes explanations of capability and maturity levels and the differences between the two CMMI representations (continuous and staged). It explains more obscure elements of CMMI, such as equivalent staging, high maturity, capability levels, and other terms and concepts that acquirers may encounter in proposals and in everyday dealings with suppliers. Finally, it cautions acquirers and users of CMMI that high maturity or capability ratings alone are not a guarantee of program success. The Guidebook, released as part of the CMMI Version 1.2 Product Suite in March 2007, is available at <www.sei.cmu.edu/publications/documents/07.reports/07tr004.html >. A Defense Acquisition University Continuous Learning Module is anticipated based on the Guidebook to help further deploy this knowledge across the workforce.

Next Generation of Improvements

The significant effort by DoD and industry to identify issues, evaluate solutions, and upgrade the CMMI to the Version 1.2 Product Suite has addressed many of the shortcomings of CMMI Version 1.1 and its application. Both model and appraisal integrity have been significantly improved. Disclosure of appraisal information has been enhanced. All CMMI training--especially lead appraiser training--has been updated, and special training with certification is required for lead appraisers conducting a high-maturity appraisal. Maturity ratings will require update after three years. Finally, the release of the Guidebook provides the acquisition community with the ability to understand and leverage the practices of their supplier's investment in process improvement.

The CMMI provides a set of best practices to be employed by the supplier. It is essential that DoD and industry use this capability in the right manner, with appropriate measure, in order to realize its benefit. Efforts for the remainder of 2007 will focus on the next generation of CMMI process improvements, which will include streamlining both the model and the appraisal method using input from a series of workshops.

Comments and questions should be addressed to [e-mail].

Schaeffer is director, systems and software engineering in the OUSD(AT & L). Osiecki is chief, RDECOM-ARDEC Armament Software Engineering Center. Richter is a senior defense analyst with the Institute for Defense Analysis. Baldwin is deputy director for software engineering and system assurance.