CMM/CMMI Level 3 or higher? No guarantee for success.

For many years, I've heard war stories about how a given supplier delivered software late, went over budget, and the quality of the product was less than expected. The people telling the stories are surprised because the supplier claimed to be a CMM [Capability Maturity Model] Level 3 or higher organization, and the clients assumed that would be a recipe for success. Now that organizations have started to migrate from CMM to CMMI [Capability Maturity Model Integration] and are achieving high CMMI levels--3 or higher--people are starting to make similar unrealistic assumptions about process maturity and project success. Why is this? What do CMMI levels really say about an organization? Could it be that the acquirers are depending too much on a "banner" and not using the information available to them to manage the project's risks, including those risks associated with using a given supplier?

What is CMMI?

The CMMI is a collection of best practices for the development and maintenance of both products and services. It was developed to enhance and replace the use of multiple process models, while preserving the government and industry investments in process improvement. By combining multiple models into a single model, the CMMI has enabled the use of common terminology, common components, common appraisal methods, and common training material across multiple disciplines. This, in turn, reduces the cost of establishing and maintaining process improvement efforts across the enterprise using multiple disciplines to deliver products or services. The CMMI currently covers systems engineering, software engineering, integrated product and process development, and supplier sourcing. The CMMI represents the consolidation of the following models:

* The Capability Maturity Model for Software (SW-CMM) v2.0 draft C

* The Systems Engineering Capability Model (SECM), also know as the Electronic Industries Alliance 731 (EIA 731)

* The Integrated Product Development Capability Maturity Model (IPD-CMM) v 0.98

In addition to being a consolidation of multiple models, the CMMI represents the incorporation of many improvements and lessons learned from earlier model use. The CMMI Framework is also consistent and compatible with the ISO/IEC 15504 Technical Report for Software Process Assessment (ISO 98).

[GRAPHIC OMITTED]

Organizations can use the model as a guide for improving their ability to develop or maintain products and services on time, within budget, and with desired quality. It provides the framework for enlarging the focus of process improvement beyond a single discipline, such as software, to improve all areas that impact product development and maintenance.

Using CMMI for Software-intensive Acquisition

A supplier's CMMI rating should be used as part only of the contract award criteria. It demonstrates simply that the supplier is capable of following mature processes, not that it necessarily will on a particular contract. As time goes on, the supplier may no longer be capable of following mature processes--thus the imposition of a three-year limit on Standard CMMI Appraisal Method for Process Improvement (SCAMPI) "A" results.

A supplier claiming to be Level 3 is no guarantee that the project within the supplier's organization is following the organization's processes. The only way an acquirer has to determine that the people actually doing the work are following mature process is to do a SCAMPI "B" or "C" assessment of the supplier. From the acquirer's perspective, SCAMPIs are used as a risk identification and mitigation tool, so they must be performed on the groups doing the acquirer's work.

[ILLUSTRATION OMITTED]

Someone once told me that without focusing on the PI--process improvement--part of SCAMPI all you get is a SCAM. Too often, acquirers demand CMMI maturity or capability levels and rely heavily upon those claims without an adequate understanding of their impact upon the work that will be performed for the acquirer. Acquirers, also, too often do not effectively utilize the SCAMPI or other appraisal methods when performing supplier monitoring and oversight. These appraisal methods allow the acquirer to tailor the appraisal scope to target specific appraisal goals and information needs in order to identify the salient risks associated with the given supplier. Those same risks, defined as weaknesses associated with individual process areas, can be tracked or monitored as the contract progresses by doing the following:

* Identifying software-related risks

* Developing a plan to mitigate the risks

* Performing trade-off analyses to establish levels of surveillance for weak areas that need improvement and critical areas where performance must be maintained

* Defining adequate reporting or insight, through the use of metrics, to be provided to the program office to facilitate continuous monitoring.

However, appraisal methods are rarely used to define the risks associated with the execution of a contract, to develop a plan to mitigate those risks, and to work the plan. A primary reason that appraisal methods like the SCAMPI are not being fully utilized by acquirers is the lack of understanding and appreciation of how an organization's process maturity and capability affects the product being developed, and how the acquirer plays a vital role in assuring that good practices are being applied by the supplier to the product being developed. Thus, SCAMPIs need to be used as input into an acquirer's risk-management process in order to fully understand the risks or weaknesses associated with the development of a particular software-intensive system.

Practice What You Preach or it Really Won't Matter

It has been shown that an acquirer with low process maturity is at greater risk of having its program delivered over cost, behind schedule, and with reduced functionality and/or avoidable defects, even if the supplier is of a higher maturity; the result is a disparity in maturity, as shown in the graphic on the previous page. For example, acquirers may try to circumvent development and management processes because they feel that following the process impacts their ability to meet the goal, resulting in rework or cost and schedule increases--which is exactly what the processes were designed to avoid in the first place.

To help the acquirer avoid such disparities, the Software Engineering Institute has developed the CMMI Acquisition Module (CMMI-AM), which defines effective and efficient practices performed by acquisition professionals in an acquisition program office. It provides a foundation for acquisition process discipline and rigor that enables product and service development to be repeatedly executed with high levels of acquisition success.

In order to avoid the feeling of being cheated or scammed, it is not enough simply to hire a supplier that claims to be of high CMMI capability or maturity. Without addressing the weaknesses of a supplier or at least taking the time to understand why they are considered weaknesses and making a conscious decision as to how to handle or not handle the weaknesses, one cannot influence the outcome or products. In addition to a supplier's capabilities and maturities, the acquirer must also perform at a high maturity--or the supplier's abilities really won't matter.

The author welcomes comments and questions. Contact him at [e-mail].

Chick works for the NAVAIR Software/Systems Support Center. He earned a bachelor's degree in computer engineering from Clemson University and a master's in Computer Science from Johns Hopkins University.