A guide to software metrics.

This guide presents an overview of the collection, analysis, and reporting of software metrics. Only the progress, effort and trouble report metrics are required for a project. However, the student should be familiar with all the metrics described below.

Software metrics are numerical data related to software development, strongly support software project management activities They relate to the four functions of management as follows:

1. Planning -- Metrics serve as a basis of cost estimating, training planning, resource planning, scheduling, and budgeting.

2. Organizing -- Size and schedule metrics influence a project's organization.

3. Controlling -- Metrics are used to status and track software development activities for compliance to plans.

4. Improving -- Metrics are used as a tool for process improvement and to identify where improvement efforts should be concentrated and measure the effects of process improvement efforts.

A metric quantifies a characteristic of a process or product. Metrics can be directly observable quantities or can be derived from one or more directly observable quantities. Examples of raw metrics include the number of source lines of code, number of documentation pages, number of staff-hours, number of tests, number of requirements, etc. Examples of derived metrics include source lines of code per staff-hour, defects per thousand lines of code, or a cost performance index.

The term indicator is used to denote a representation of metric data that provides insight into an ongoing software development project or process improvement activity. Indicators are metrics in a form suitable for assessing project behavior or process improvement. For example, an indicator may be the behavior of a metric over time or the ratio of two metrics. Indicators may include the comparison of actual values versus the plan, project stability metrics, or quality metrics. Examples of indicators used on a project include actual versus planned task completions, actual versus planned staffing, number of trouble reports written and resolved over time, and number of requirements changes over time.

Indicators are used in conjunction with one another to provide a more complete picture of project or organization behavior. For example, a program indicator is related to requirements and size indicators. AU three indicators should be used and intepreted together.

Metrics Set

The defined indicators are consistent with the Software Engineering Institute's Capability Maturity Model (CMM). Table I shows the indicator categories, the management insight provided, and the specific indicators for recommended metrics. Depending upon the nature of the project, specific contractual requirements, or management preference, a project may choose to collect additional metrics or to tailor the recommended set.

Chart Construction Summary

Charts are prepared for the standard metrics. All charts require titles, legends, and labels for all axes. They should clearly and succinctly show the metrics of interest with no excessive detail to detract the eye. Do not overuse different line types, patterns, or color, or added dimentionality unless used specifically to differentiate items. Overlayed data is preferable to multiple charts when the different data are related to each other and can he meaningfully depicted without obscuring other details.

The most common type of chart is the tracking chart. This chart is used extensively for the Progress indicator, and is used in similar forms for many of the other indicators. For task progress, it depicts the cumulative number of planned and actual task completions (or milestones) against time. For other indicators, it may show actual versus planned staffing profiles, actual versus planned software size, actual versus planned resource utilization or other measures compared over time.

There are many ways to modify the tracking chart. A horizontal planned line representing the cumulative goal can be drawn at the top, multiple types of tasks can be overlaid on a single tracking chart (such as design, code, and integration), or the chart can be overlaid with other types of data.

It is recommended that tracked quantities be shown as a line chart, and that replanned task progress be shown as a separate planning line. The original planned baseline is kept on the chart, as well as all replanning data if there is more than a single replan.

The following sections provide brief descriptions of the different metrics categories with samples of the required charts. Individual projects may enhance the charts for their situations or have additional charts for the categories.

Progress

Progress indicators provide information on how well the project is performing with respect to planned task completions and keeping schedule comniitments. Tasks are scheduled and then progress is tracked to the schedules. Metrics are collected for the activities and milestones identified in the project schedules. Metrics on actual completions are compared to those of planned completions to determine whether there are deviations to the plan. The difference between the actual and planned completions indicates the deviations from the plan.

Each project identifies tasks for which progress metrics will be collected.

The completion criteria for each task must be well defined and measurable. The project should establish range limits (thresholds) on the planned task progress for the project. The thresholds are used for management of software development risk.

Figure 1. Depics the cumulative number of planned and actual completions (or milestones) over time. Note that this chart is generic, and each project will substitute specific tasks (units, milestones, SLOCS, etc.) Additionally, each project is expected to produce multiple progress charts for different types of tasks, different teams, etc.

Effort

Effort indicators allow the software manager to track personnel resources. They provide visibility into the contribution of staffing to project costs, schedule adherence, product quality and the amount of effort required for each activity. Effort indicators include trends in actual staffing levels, staffing profile by activity or labor category, or a profile of unplanned staff losses.

Effort indicators may be used by all levels of project software management to measure the actual profile against the plan. Each level of management forms a profile for its area of control and monitors the actual profile against the plan.

Determining the number of staff needed at any one time is an important function performed by software management. By summing the number of staff during each reporting period, the composite staffing profile for the project can be determined. These indicators are applied during all life-cycles phases, from project inception to project end. Effort metrics are to be collected and reported at least on a monthly basis. The effort and cost metrics are related. By convention, effort metrics are non-cumulative expenditures of human resources, and cost metrics are cumulative levels of effort as tracked by earned value. Thus, cost metrics are a cumulative depiction of effort.

Figure 2 shows a sample plot of monthly actual versus planned effort

Review Results

Review Results indicators provide insight into the status of action items from life-cycle reviews. The term Action Item (Al) refers to inspection defects and customer comments. Reviews include the following:

* Formal inspections of software documents or code

* Formal customer milestones, e.g., SSR, PDR, CDR, or TRR 

* Informal peer evaluations of products, e.g., walkthroughs, technical reviews, or internal PDRs

* Management reviews

* Process reviews, e.g. SQA audits, SEI CMM assessments, or the causal analysis from formal inspections.

There are standards for some reviews, as well as procedures for conducting them. For example, formal inspections result in assertion logs that document the minor and major defects uncovered by the inspection process. Therefore, standard review result indicators for formal inspections are-.

1. Counts of minor/major defects

2. Rates of defect detection (e.g., assertions per inspection meeting minute, defects per inspected document page, or defects per KSLOC of code inspected)

3. Defect status (e.g., age of open defects, number of open/closed defects, and breakdown by defect categories).

A customer-conducted review such as a Preliminary Design Review (PDR) generates AIs that must be closed before approval of the Software Design Document. Therefore, standard review result mchcators for a PDR are the number of comments written and their status (open, closed, and age).

Review metrics record the AIs identified in the review findings and track them until they are resolved. These metrics provide status on both products and processes. Review results are not to be used to evaluate the performance of individuals.

Review Results are collected and reported at least monthly at every stage of the software life cycle, but preferably weekly for key AIs.

Trouble Reports

TR indicators provide managers with insight into the quality of the product, software reliability, and the effectiveness of testing. They also provide information on the software development process. The terms defect and problem will be used interchangeably herein. Monthly tracking of TR indicators shows the projects trends in the following areas:

1. The rate at which TRs are being written and resolved.

2. The type and severity of the TRs.

3. Relationship between the number of TRs and the number of test cases passed or the number of test steps passed.

4. The TR density (the number of TRs per unit size).

5. The number of defects in each software application/unit.

TR indicators are applicable only in the following life cycle stages (and each release of the software within these stages, and during the informal and formal test segments of these stages) (1) application test and integration, (2) system test, (3) acceptance test. Thus the TR indicators are applicable only to defects during the operation or execution of a computer program. Due to the shortness of testing periods, and the dynamics involved between the test team and the implementation team that analyzes the TR's and fixes the defects, the TR indicators are generally evaluated on a weekly basis.

The terms open and closed are defined as follows:

Open: The problem has been solved.

Closed: The investigation is complete and the action required to resolve the problem has been proposed, implemented and verified to the satisfaction of all concerned. In some cases, a TR will be found to be invalid as part of the investigative process and closed immediately.

Requirements Stability

Requirements Stability provides an indication of the completeness, stability, and understanding of the requirements. It indicates the number of changes to the requirements and the amount of information needed to complete the requirements definition. A lack of requirements stability can lead to poor product quality, increased cost, and schedule slippage.

Requirements stability indicators are in the form of trend charts that show the total number of requirements, cumulative changes to the requirements, and the number of TBDs over time. A TBD refers to an undefined requirement. Based on requirements stability trends, corrective action may be necessary.

Requirements stability is applicable during all life-cycles phases, from project inception to the end. The requirements stability indicators are most important during requirements and design phases. Requirements stability metrics are collected and reported on a monthly basis.

Figure 6 shows an example of the total number of requirements, the cumulative number of requirements changes, and the number of remaining TBDs over time. It may be desirable to also show the number of added, modified and deleted requirements over time.

Size Stability

Software size is a critical input to project planning. The size estimate and other factors are used to derive effort and schedule before and during a project.

The software manager tracks the actual versus planned software product size. Various indicators show trends in the estimated code size, trends by code, the variation of actual software size from estimates or the size variation by release.

Size stability is derived from changes in the size estimate as time goes on. It provides an indication of the completeness and stability of the requirements, the understanding of the requirements, design thoroughness and stability, and the capability of the software development staff to meet the current budget and schedule. Size instability may indicate the need for corrective action.

Size metrics are applicable during all life-cycle phases. Size metrics are collected and reported on a monthly basis, or more often as necessary.

Figure 7 shows an example of planned and currently estimated software size per release over time. Besides showing re--allocation of software content between releases, this also shows the growth in the total estimated size.

Computer Resource Utilization

Computer Resource Utilization indicators show whether the software is using the planned amount of system resources. The computer resources are normally CPU time, 1/0, and memory. For some software, the constraints of computer resources significantly affect the design, implementation, and testing of the product. They can also be used to replan, re-estirnate, and guide resource acquisition.

Computer resource utilization is planned during the requirements activity and reviewed during the design activity. Resources are monitored from the start of implementation activity to the end of the life cycle.

For memory utilization, the unit of data is the byte, word, or half-word. For CPU time, the unit of data is either MIPS (millions of instructions per second), or the percentage of CPU time used during a peak period. For 1/0 time, the unit of data is the percentage of 1/0 time used during a peak period.

Resource Utilization data is collected and reported at least monthly, with the period between collection and reporting becoming shorter as the software system nears completion and a better picture of software performance can be seen. Note that the resource utilization is normally an estimate until integration occurs, at which time the actual data is available.

Figure 8 shows an example of CPU and memory use as a percent of available and the maximum allowed.

Training

Training indicators provide managers with information on the training program and whether the staff has necessary skills. A trained staff is a commitment-The manager must ensure that the staff has the skills needed to perform their assigned tasks. The objective of the training indicator is to provide visibility into the training process, to ensure effective utilization of training, and to provide project software managers with an indication of their staffs skill mixture.

The manager should investigate the deviations in the number of classes taught from the number of classes planned, and the deviation of the number of staff taught to the planned number. The quality of the training program should also be determined from completed course evaluation sheets. The number of waivers requested and approved for training should also be tracked.

Figure 9 shows a sample graph of the total monthly attendance of personnel attending training classes. It represents the sum of the number of personnel attending all classes.

Overview of Project Procedures

When starting a software development project, determine the list of software metrics. Use the goal-question-measure paradigm to select appropriate measurements for the project. To do this, first establish goals for the project, develop questions to be answered by measurement, and then collect the data needed. Document the software metrics to be collected for the project in the project plan. During a project, collect and report the metrics. Use the collected metrics data to track-project, status. At the end of a project utilize the data for postmortem reporting.

Use automated data collection and analysis tools whenever possible to collect the metrics data for your project Some tools that can aid in this process include Amadeus for code measurement, TR statusing tools, requirements management tools, etc.