Improving the integrity of the CMMI Product Suite.
DoD and industry have initiated various efforts to better understand the issues contributing to the difference in expectations versus DoD's observations. First, the National Defense Industrial Association (NDIA) Workshop and Summit on CMMI Use in Acquisition, held in September 2005, identified issues and offered a set of preliminary recommendations. The initial set of issues that DoD and industry determined to require further attention were:
* Program office understanding of the need for and benefit of mature and capable processes
* Lack of training and guidance for acquiring organizations on CMMI usage
* Limited content and usefulness of the appraisal disclosure statements (ADSs) submitted by suppliers
* Benefits and drawbacks of specifying or requiring minimum CMMI maturity level ratings in RFPs
* Organizational approach to CMMI implementation and appraisals.
In the spring of 2006, a government review and assessment team validated these issues, refined and augmented them, then recommended actions to the CMMI Steering Group, which is composed of representatives from DoD, industry, and sponsors. Finally, the Defense Contract Management Agency conducted a data call that confirmed their initial assessment about inconsistencies between appraised ratings claimed and observed program performance.
These efforts resulted in a consolidated list of issues that the sponsors believed needed to be effectively addressed:
* Observations that suppliers execute at lower maturity levels than they have achieved in a formal appraisal
* Once achieved by an organization, a CMMI maturity level rating existed for life
* Organizations do not necessarily incorporate CMMI-appraised processes on new projects
* CMMI-driven processes and practices are not consistently applied at the project level after contract award
* Appraisal sampling procedures did not ensure adequate coverage of the organizational unit granted the appraisal rating
* Appraiser quality and training raised issues with integrity of maturity ratings
* Lack of consistent understanding and application of high-maturity practices
* Lack of definition within the CMMI Product Suite on what constitutes high maturity
* Content of ADSs lacks specificity and is not useful to program offices
* Inadequate training and education for acquirers.
CMMI Version 1.2 Release
The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, and the NDIA Systems Engineering Division--the co-sponsors of the CMMI--directed the CMMI Steering Group to take the necessary actions to improve the integrity of appraisal results and address other shortcomings in the CMMI Product Suite. Implementation teams under the stewardship of the Steering Group developed the necessary changes to the model, the appraisal method, and the training with this goal in mind. As a result, the CMMI Product Suite has been modified to address each of the identified issues.
"Level for Life" Eliminated
Many organizations have attained a CMMI maturity rating over the years. Often these ratings are prominently displayed in a firm's marketing material and highlighted in proposals. Since there was no definitive rule regarding when appraisal results became invalid, it was left to interpretation when appraisal results were considered stale. Version 1.2 established that appraisal results would remain valid for a maximum of three years. Organizations will have to re-appraise using the current CMMI model and appraisal method in order to make any maturity or capability level claims beyond this.
CMMI Version 1.1 Retired
The entire CMMI Version 1.1 Product Suite will be retired effective Aug. 31, 2007. Thus, all appraisals after that date must use the Version 1.2 model and appraisal method. The full set of CMMI Version 1.2 policies is available at <www.sei.cmu.edu/cmmi/appraisals/cmmiv11-sunsetappraisal-policies.html >.
Organizational Processes Applied at Program Start-up
Organizations at maturity Level 3 or higher are expected to employ their organizational processes on new programs. There is evidence that some organizations either do not employ them or delay applying them on new programs. Material added to the model in Version 1.2 now requires organizations to deploy their set of standard processes at project start-up and deploy changes to those processes as appropriate throughout the life of the project.
Appraisal Disclosure Statement Improved
The ADS was enhanced by the addition of specific information on the sampling approach, percentage of projects sampled, and the organizational scope. The updated Version 1.2 ADS requires provision of detailed information on what was appraised in an organization. Previously, it was often not clear which organizational unit actually had attained the maturity rating in a large corporation. The ADS will now list specific projects, organizational units, and domain information. This provides insight to an acquirer who is interested in the capability of a particular organization for a specific project, unit, or domain. Additionally, these changes in the ADS are intended to improve program selection integrity by placing more responsibility on lead appraisers to ensure a representative organizational sample. New, required disclosure information regarding applicability to the organizational unit clarifies this issue.
Verification of Level 4 and Level 5 Processes
To achieve a high maturity rating, an organization must quantitatively manage (maturity Level 4) or optimize (maturity Level 5) select processes and subprocesses. Review of CMMI appraisals revealed a lack of consistency in what constituted Levels 4 and 5. Some organizations would select one subprocess, while others would attempt to quantitatively manage and optimize all of their processes. Neither behavior exhibits the intent of the model, which is to apply high maturity practices on select processes or subprocesses that are important to an organization's business goals and objectives. Some processes may not warrant high maturity application. Further, if the chosen processes or subprocesses are not those of value to the organization, domain, or program, the supplier's high maturity activity may not add benefit. With Version 1.2 of the ADS, lead appraisers now have the added responsibility to verify that the chosen processes relate to the organization's business objectives. Additionally, the Software Engineering Institute (SEI), designated by the sponsors as the CMMI Product Suite steward, has undertaken a significant effort to set qualifications for high maturity appraisers, update high maturity training with certification, and deploy these standards across the pool of high maturity appraisers.
Appraisal Results Reporting Changes
Acquirers should have insight into all the process areas that make up an organization's capability or maturity level ratings. A capability level profile provides the capability levels of the individual CMMI process areas. Examination of the capability level profile provides the ability to determine whether an organization is mature in processes that are critical or relevant to an acquirer's program. A single maturity level rating does not provide this insight. In this regard, the capability profile level is more meaningful and is endorsed by the DoD as a better metric to understand process capability than relying on a single maturity level rating.
Quality Audit Review
The SEI, the CMMI steward, must review and accept results before appraised organizations will be allowed to publicly announce the appraisal results or use the information in a formal proposal to the government or other acquirer. This allows the SEI to identify and correct any inconsistencies in the appraisal process before the results are declared official.
DoD Contractor ADSs Posted
Firms that contract with the DoD and seek to have a recent CMMI appraisal considered are now required to post their ADS on the SEI's Published Appraisal Report Site (PARS) at <http://sas.sei.cmu.edu/pars/pars.aspx> for government acquirer review. This reporting was formerly optional, at the behest of the contractor. Acquirers are now encouraged to check the PARS site to validate a supplier's maturity level claim, since only validated results are posted on PARS. In addition, if the appraisal was conducted after Nov. 1, 2006, acquirers will be able to gain the appraisal specifics available in Version 1.2 of the ADS, since the use of the new ADS is mandated as of that date.
Lead Appraisers to be Independent
Lead Appraisers from the same organization as the one sponsoring the formal appraisal can no longer grant a capability or maturity level rating to their organization. It was determined by the CMMI Steering Group and sponsors that this independence is necessary to ensure integrity of appraisal results. The requirement applies only to the appraisal lead and not to the members of the appraisal team.
Guidebook for Acquirers Published
Understanding and Leveraging a Supplier's CMMI Efforts: A Guidebook for Acquirers was a major effort to help acquirers benefit from a supplier's use of CMMI while avoiding the pitfalls associated with unrealistic expectations. The Guidebook describes CMMI fundamentals and much of the information summarized in this article to help acquirers effectively use information obtained from a supplier's CMMI effort. It includes explanations of capability and maturity levels and the differences between the two CMMI representations (continuous and staged). It explains more obscure elements of CMMI, such as equivalent staging, high maturity, capability levels, and other terms and concepts that acquirers may encounter in proposals and in everyday dealings with suppliers. Finally, it cautions acquirers and users of CMMI that high maturity or capability ratings alone are not a guarantee of program success. The Guidebook, released as part of the CMMI Version 1.2 Product Suite in March 2007, is available at <www.sei.cmu.edu/publications/documents/07.reports/07tr004.html >. A Defense Acquisition University Continuous Learning Module is anticipated based on the Guidebook to help further deploy this knowledge across the workforce.
Next Generation of Improvements
The significant effort by DoD and industry to identify issues, evaluate solutions, and upgrade the CMMI to the Version 1.2 Product Suite has addressed many of the shortcomings of CMMI Version 1.1 and its application. Both model and appraisal integrity have been significantly improved. Disclosure of appraisal information has been enhanced. All CMMI training--especially lead appraiser training--has been updated, and special training with certification is required for lead appraisers conducting a high-maturity appraisal. Maturity ratings will require update after three years. Finally, the release of the Guidebook provides the acquisition community with the ability to understand and leverage the practices of their supplier's investment in process improvement.
The CMMI provides a set of best practices to be employed by the supplier. It is essential that DoD and industry use this capability in the right manner, with appropriate measure, in order to realize its benefit. Efforts for the remainder of 2007 will focus on the next generation of CMMI process improvements, which will include streamlining both the model and the appraisal method using input from a series of workshops.
Comments and questions should be addressed to ATL-SSA@osd.mil.
Schaeffer is director, systems and software engineering in the OUSD(AT & L). Osiecki is chief, RDECOM-ARDEC Armament Software Engineering Center. Richter is a senior defense analyst with the Institute for Defense Analysis. Baldwin is deputy director for software engineering and system assurance.
|Printer friendly Cite/link Email Feedback|
|Author:||Schaeffer, Mark D.; Osiecki, Lawrence T.; Richter, Karen; Baldwin, Kristen|
|Publication:||Defense AT & L|
|Date:||Jul 1, 2007|
|Previous Article:||The Process Cycle.|
|Next Article:||Source selection: communicating with offerors.|