Improving the reliability & validity of independent government cost estimates for information technology service.
The Federal Acquisition Regulation (FAR) requires an IGCE for every procurement valued above the simplified acquisition threshold (FAR 2.101). The United States Army Network Enterprise Technology Command (NETCOM) requires an IGCE for all contract requirements, independent of cost, as part of its internal cost control program.
NETCOM Comptroller Decision Support Team
Within the NETCOM, our Decision Support Team's primary functions are to review, validate, control, and register cost estimates and submissions and to establish command cost positions. This team also is the proponent for regulations pertaining to cost, cost control and cost estimates.
We employ three levels of review and validation for cost estimates. Each level prescribes a greater depth of analysis.
Level I requires the greatest depth of analysis and is for cost estimates exceeding $2 million in total life-cycle costs. Associated costs may include research and development (R&D), investment, annual recurring costs, or savings. These initiatives will contain a statement that alternatives have been considered. We also require Level I validation and review for projects or acquisitions intended for implementation throughout NETCOM. These include all command-wide office automation projects and initiatives managed and controlled by NETCOM, regardless of the cost of those programs and systems, as well as those projects designated by the Commanding General, NETCOM for intensive life-cycle management and decision review. Level 1 validation requires three signatures: the preparer (typically a GS-9 through GS-11), a reviewer (typically a GS-12 or GS-13), and an approver (typically a GS-13 to GS-15).
Level II applies to cost estimates valued between $1 million and $2 million in total R&D costs, investment costs, annual costs, or savings. Level II cost estimates are reviewed by a GS-9 to GS-12 Cost Analyst to ensure:
* Calculation of costs and cost factors are documented
* Assumptions and constraints are logical and reasonable
* Cost estimates are reproducible, internally consistent, and mathematically correct
* All appropriate cost factors are included
* Identification of source documents and dates published
* Estimates track to the source document
* Source documents are current
* Latest inflation indices have been used
* Any deviations from approved source documents are adequately explained
* Alternatives have been considered where appropriate.
Level III cost estimates are for requirements valued at $1,000,000 or less. Level III cost estimates are reviewed by a GS-9 to GS-13 Cost Analyst to ensure:
* Assumptions are logical and reasonable
* All appropriate costs factors are included.
By the way, NETCOM has established performance standards for its cost analysts. These are set at greater than 80% reliability for GS-9 to GS-11, greater than 85% reliability for GS-12, and 90% or better reliability for GS-13 and above.
A NETCOM regulation prescribes the format of the IGCE. We use an electronic workbook with pre-established rates for indirect costs including labor overhead, material overhead, G&A expenses, and profit or fee. Indirect costs may also include Workman's Compensation and Liability, State Unemployment Insurance, Federal Unemployment Insurance, Social Security (FICA), and Payroll Taxes. Variation In the indirect rates is allowed if it can be sufficiently documented and contributes to an improvement in the overall estimate.
Typically, the organization that generates the requirement (the "requiring activity") uses fully loaded labor rates obtained from pre-negotiated Indefinite Delivery/Indefinite Quantity contracts or General Service Administration schedules. The decision support analyst who creates the IGCE uses wages reported either from the Bureau of Labor Statistics (BLS) or Salary.com. Each analyst has been assigned to use one of these sources exclusively when preparing IGCEs.
Within the Information Technology (IT) mission area, we have multiple labor categories from which to select. The validity of each estimate is highly dependent on selecting the appropriate labor category or job title. Table 1 shows BLS median nationwide hourly salary rates for the associated occupational codes compared with corresponding job titles on Salary.com. It is important to note that the BLS occupational codes are associated with a collection of job titles. For example, 11-3021- Computer and Information Systems Managers includes job titles such as: IT Manager, IT Director, Information System (IS) Director, Data Processing Manager, Management Information Systems Director, IS Manager, IS Supervisor, Computing Services Director, Director of Application Development, and Technical Services Manager. In contrast, the free Salary.com salary wizard produces the median nationwide salary for each given job title. Both sources of salary information report rates on national, regional, and metropolitan levels of detail.
As mentioned previously, selecting the appropriate job category is vital to creating a valid estimate. Often, we are able to match job titles provided by the requiring activity directly with BLS labor categories or Salary.com job titles. In many cases, however, there is not a direct match, thus introducing potential error in the IGCE. To minimize error, the Decision Support analyst studies the Performance Work Statement submitted by the requiring activity to better understand mission requirements. Keeping mission requirements in mind, he or she then makes a judgment as to which labor category or job title best fits the requirement.
To determine whether one wage source produces more valid estimates, we collected data on 14 completed estimates. We used BLS as the rate source for six of the 14 and Salary.com for the remaining eight estimates. Upon initial review, it appeared that BLS labor rates led to higher validity in the IGCEs; however, the estimates had been completed by two different analysts, thereby leading to an increased degree of variability and the possibility of analyst bias. To reduce variability in the data, each of the 14 estimates was redone by one analyst twice--once using BLS labor rates and once using Salary.com labor rates. The results, compared to government cost estimates (GCEs) are displayed as scatterplots in Figures 1 and 2, below.
Figures 1 and 2 illustrate that, keeping the analyst constant and using wages from BLS and Salary.com respectively, produce equally reliable estimates.
Establishing internal controls is foundational to producing reliable and valid IGCEs. Command-wide standardization of the indirect cost factors used helps reduce the variability in estimates. This standardization also aids in identifying the source of variation when it occurs. For example, the cause of the variation in this study, originally assumed to be the labor rate source, was actually analyst bias. This finding yielded valuable insight that helped us improve the IGCE process. Since the Decision Support team generates many estimates with the same or similar IT labor categories, the push towards standardizing direct costs (in the same way indirect costs have been standardized) eliminates or greatly reduces analyst bias and leads to more accurate estimates. And, standardizing direct costs also streamlines the process, thereby increasing analyst efficiency.
Although this study focused on the Decision Support team and our estimates for IT services, the findings can be applied across the Department of Defense. By standardizing both indirect and direct costs within routinely performed estimate calculations, one can increase the validity of each estimate and lead to a more efficient workforce.
Acknowledgement: The authors wish to thank the following NETCOM Decision Support Team members for their contributions: Mr. Tyrel Paradis, Mr. Brian Clark, Mr. Jeraco Speelman and Ms. Jane Gallant.
MS. HEATHER WACLAWIK
Ms. Heather Waclawlk is an Operations Research Systems Analyst with the NETCOM Decision Support team. She holds a BS degree in Mathematics from the University of Arizona and has been employed by NETCOM since December of 2011. She is a member of the ASMC's Cochise Chapter.
MR. LAWRENCE COUTURE
Mr. Lawrence Couture is the Chief of NETCOM's Program Analysis and Evaluation Division, Decision Support team. He holds a BA degree from the University of Louisville and a Masters degree from Spalding University. He has been employed by NETCOM as a Senior Operations Research Systems Analyst for 14 years and is a member of the ASMC's Cochise Chapter.
Table 1. Comparison of rates associated with labor categories versus specific job titles. Occupational Code and Description BLS Job Title Salary.com 11-3021--Computer and $58.15 Applications $51.89 Information Systems Development Manager Managers 15-1120--Computer and $38.68 Applications Systems $32.38 Information Analysts Analyst II 15-1121--Computer $38.31 Applications Systems $32.38 Systems Analysts Analyst II 15-1122--Information $41.43 Systems/Application $38.01 Security Analysts Security Analyst 15-1131--Computer $35.71 Applications $51.31 Programmers Programming Manager 15-1133--Software $47.59 Software Developer $36.68 Developers, Systems II Software 15-1134--Web Developers $30.05 Web Applications $37.97 Developer II 15-1151--Computer User $22.32 Technical Support $19.08 Support Specialists Representative II 15-1152--Computer $28.41 Network $31,18 Network Support Administrator II Specialists 17-2051--Computer $48.52 Hardware Engineer II $34.27 Hardware Engineers