An introduction to government technical evaluations: a contracting officer's perspective; Comprehensive, accurate technical evaluations are the foundation for negotiating a fair and reasonable modification to any acquisition contract.
The technical evaluation of the contractor's proposal directly affects the outcome of the negotiation stage of the acquisition process. In my experience as a contracting officer (CO), these evaluations have been sorely lacking content and credibility, which may be due to a shortage of experienced personnel or availability of historical data. Regardless of specific reasons, it is time for us to focus as an acquisition community on the technical evaluation of the contractor's proposal--using all means to improve it and realize the benefits for the services and supplies the government acquires daily.
The Federal Acquisition Regulation (FAR) requires that each contractor's proposal be analyzed to assure that the price paid by the government for goods or services is fair and reasonable. FAR 15.404-1 (e) defines technical analysis as follows:
(1) The contracting officer may request that personnel having specialized knowledge, skills, experience, or capability in engineering, science, or management perform a technical analysis of the proposed types and quantities of materials, labor, processes, special tooling, facilities, the reasonableness of scrap and spoilage, and other associated factors set forth in the proposal(s), to determine the need for and reasonableness of the proposed resources, assuming reasonable economy and efficiency.
(2) At a minimum, the technical analysis should examine the types and quantities of material proposed and the need for the types and quantities of labor hours and the labor mix. Any other data that may be pertinent to an assessment of the offeror's ability to accomplish the technical requirements or to the cost or price analysis of the service or product being proposed should also be included in the analysis.
Purpose and Use
The purpose of the technical evaluation is to determine whether the contractor's proposed expenditure of labor and resources relates to the performance promises and schedule objectives of the contract. It addresses the quantitative and qualitative aspects of the proposed direct charges, including labor, materials, subcontracts, interdivisional work, computer usage, travel, and other direct costs.
The technical evaluation is the assessment of the proposed effort to accomplish the contract requirements specified in the request for proposal (RFP). It is not an evaluation of the dollar amounts but rather a technical analysis of the rationale, estimates, and information behind the dollar amounts. In the sole-source environment, the technical assessment supplements a cost analysis of the specific cost elements and tasks.
The Defense Contract Audit Agency (DCAA), Defense Contract Management Agency (DCMA), or the program's cost and price analyst usually performs the cost analysis in accordance with FAR 15.404-1(a) (5), which states, "The contracting officer may request the advice and assistance of other experts to ensure that an appropriate analysis is performed." The division of labor is usually divided between the field organization (DCMA/DCAA) and the program office. The program office personnel usually look at engineering labor hours, as well as material quantities, subcontract labor, travel, and other direct costs, while DCMA looks at labor hours for testing, production, tooling, quality control, and planning. DCAA reviews any subcontracts, materials, other direct costs, and rates and factors for the proposal.
The technical evaluation is necessary when the program office requires cost or pricing data. The completeness and accuracy of the technical evaluation is necessary to prepare the government's negotiation position. While the contract negotiator and price analyst prepare the negotiating position, the program office will participate in all clearance briefings to support the findings of the technical evaluation. The evaluation supports the government's position in the event of disputes or defective pricing allegations.
The technical evaluation provides the program office the opportunity to verify the contractor's understanding of the government's technical requirements. The technical evaluation is used to assess the risk of meeting the performance parameters established by the government. The technical evaluator is accountable for documenting measurable assessments to the realism of the contractor's proposed labor and material budgets.
Upon receipt of the contractor's proposal, the program's technical evaluator performs a preliminary review of the proposal to ensure at least the following recommended compliances:
(1) The overall technical proposal is adequate and addresses scope, deliveries, and schedules required by the request for proposal (RFP);
(2) The contractor's technical approach in the proposal is based upon sound engineering concepts;
(3) The contractor provides adequate information to support specific quantities of labor and materials;
(4) The contractor's proposal contains adequate description of its basis for direct labor hours, including sufficient rational for engineering judgment and projections from prior work completed on similar programs;
(5) The contractor provides adequate explanation for factored labor hours;
(6) The contractor's proposal provides supporting data to justify proposed material, scrap, rework, attrition, or other factors; and
(7) The contractor provides reasonable rationale for special tooling and test equipment to establish that the proposed items are required for the program.
Factors that should also be noted in the technical evaluation include the complexity of the project and associated risks of satisfactory technical performance, the contractor's proposed man-loading analysis, the accounting and estimating methodology, relevant technical and management experience levels, and the contractor's judgment on technical issues.
The failure to provide a comprehensive technical analysis of a contractor's proposal can seriously delay the award of the contract action, impede an effective price analysis, extend negotiations, and cause difficulties in contract performance after award. Therefore, it is essential that the technical engineer be specific in analyzing the contents of the contractor's proposal, so the evaluation is consistent, complete, and accurate to withstand contractor scrutiny.
During negotiations, it will be the contractor's responsibility to look at this evaluation, questioning the engineering judgment, the methodology of the technical analysis, and the conclusion of the technical assessment. Without solid substantiation, the evaluation becomes the weak link in the negotiation process--it can be difficult to substantiate the technical evaluation because of inexperienced evaluators lacking relevant engineering experience, time constraints, and hard-to-access historical data.
To assist the evaluator in his/her assessment, there are five key elements for a successful technical evaluation: (1) organization, (2) preparation, (3) communication, (4) focus, and (5) consolidation of the elements into a cohesive analysis useful during negotiations.
The organization of the technical evaluation must be parallel with the contractor's technical proposal, for ease of understanding the differences (in rates or factors) between the parties and preserving the task segregation for labor and material. By keeping a strict adherence to organization of the technical assessment consistent with the contractor's proposal, the technical evaluation will assist the price analyst and negotiator to establish a reasonable position based on any measurable, easily recognizable differences.
A primer for organizing a technical evaluation consists of starting with a summary paragraph, briefly describing the work and any unique aspects to the contractor's proposal. The technical evaluation should then summarize the proposed hours with clarity, traceability, and completeness. An explanation for all disallowed hours or recommended reductions should reference the proposal documentation. The technical evaluation should document any agreements on tasks or costs as a result of fact-finding. Finally, the technical evaluation should conclude with recommendations and direct traceability to the task sheets.
The bases for understanding and producing a cohesive technical evaluation are (a) reading and understanding the SOW and the contractor's cost and technical proposal; (b) reviewing the scope of the tasks proposed by the contractor; (c) reviewing the contractor's estimating methodologies and rationale; (d) evaluating the detailed estimates, calculations, and factor applications used in the contractor's proposal; and then (e) formulating a technical analysis plan.
To understand the proposal, the technical engineer must know the requirements as they are written in the SOW. The most common source of inadequate technical evaluations is the lack of understanding of key issues of the proposal and not understanding the government's requirements. Therefore, it is vital that there are valid government requirements for the contractor's proposal. In many cases, the requirements have not been determined and the basis for fact-finding has been the determination of a valid government requirement for the acquisition. Any changes to the SOW, then, must be provided to the CO for incorporation into the final contract documents.
The evaluation should also address any open technical issues that still need to be resolved during further discussions. The technical evaluator should comment on the reasonableness of the proposed material, including such factors as the kind and quantities and necessity of purchasing such material for the project, while the DCAA auditor evaluates the rates of the contractor's proposal and quotes for the material. Similarly, for labor assessments, the evaluator should evaluate the types, quantities, and skill mixes of labor hours proposed by the contractor and provide a detailed estimate of the necessary labor hours and mix, as well as the rationale for differences with the contractor's estimate to accomplish the proposed work.
Finally, to formulate a comprehensive technical analysis plan, the evaluator will prioritize tasks and look for common threads in the proposal for consistency of purpose and understanding. Once the requirements are known, the cost proposal is fully understood and the technical analysis plan for evaluation is set, the evaluation can proceed to the next important element.
If additional information is required to understand the contractor's cost and technical proposal, the program's technical evaluator should contact the CO to request better insight into the proposal. As a result, a series of fact-finding meetings may be necessary to resolve any issues and gain a full understanding of what the contractor is proposing for the contract.
Fact-finding usually consists of a multi-disciplinary team jointly led by the buyer and project officer--this meeting aims to reconcile assumptions, facts, and judgments associated with the contractor's proposal and gain further insight to the preparation and examination of the proposal. It is important to have multi-disciplinary buy-in from all functionals in the program office for the fact-finding meetings. These meetings should be arranged early in the process--so as not to impact the overall acquisition schedule--and should focus on gaining insight into the rationale behind the contractor's proposed effort.
The key element in fact-finding is to understand how to communicate and investigate the methodology of the proposal. As a result of fact-finding, the contractor may submit a revised proposal. The revision may result in the need for a secondary technical evaluation, which would include an assessment of the revised proposal. If there are clarifications of ground rules or assumptions required from the contractor during fact-finding, the technical assessment should explain any assumptions made in the technical evaluation of the proposal, based on the clarifications provided by the contractor.
Focus: Maximum Investment
The technical evaluation must be able to stand as an independent document during negotiations, substantiating the government's position for exceptions to the contractor's proposal. The project office should discuss any divergences in the government estimate in terms of hours, skills, or materials.
It is important for the technical evaluation to answer the question: "What is the basis for the difference?" The evaluation should be substantiated with facts and also explain the differences in estimates. Without clear rationale, the contractor will question the authority of the technical evaluation during negotiations. Finally, it should explain clearly the impact of accepting the proposed or government recommended hours, skills, or materials in terms of program risk, cost, or schedule impact, as well as any oversight necessary to sustain the proposed effort.
Consolidation of Elements
The technical evaluation should be summarized to the level of detail commensurate with the proposal structure--coordinated within the program office and with the price analyst--and with the objective to assist contract negotiations. It should contain evaluation sheets of all separately identified technical and administrative tasks, line items, and milestones. The evaluation sheets (see Figure 1 on page 42) should detail (a) the basis of the contractor's estimate, (b) the basis of the government's objective position, and (c) the basis of the government's going-in position.
The technical evaluation must be clear and show the reference page in the proposal, the cost element, the proposed amount, the recommended position, and the explanation for the recommendation or exception. A summary per element can be included to assist the government price analyst and contract negotiator in formulating a negotiation position. The rationale will provide arguments to support the recommended position by using such methods as comparison with similar proposals, catalog or market-price information, established industry standards, engineering judgment, or any other supportable methodology. This is key for the technical evaluator because without a focus on traceability and concise evaluation support, the technical evaluation will need numerous revisions before the CO considers it adequate.
The technical evaluation is written by the program's lead evaluator and reviewed by the program office's division level engineer. The division level engineer signs a cover letter to the contracting officer, certifying the completeness and accuracy of the technical evaluation. Accountability lies with the program office to write a concise technical evaluation that assesses the contractor's proposed performance, schedule, and cost. Figure 2 (on this page) gives an example of a technical evaluation.
Writing technical evaluations involves a review of the contractor's methodologies and rationale. A contractor's methodology is the logic used to develop the proposed hours. The rationale allows the technical engineer to understand the methodology the contractor used for developing their position. These may consist of averages, cost-estimating relationships, manpower analysis, skill mixes, man-loading estimates, judgmental estimates, and unsupported estimates, if not already covered in the audit report or price analysis report. The technical evaluator also needs to understand things from a government perspective.
Averages. The use of an "average" by the contractor sounds reasonable but may not be appropriate in all circumstances. Is the average based on similar elements? The data must be checked for accuracy and completeness and variations in the data points. A trend analysis may also be appropriate when reviewing averages. Is the proposed effort consistent with the trend analysis? The bottom line for the evaluator is to consider that the government is concerned with the specific effort, not with an average number of labor hours.
Cost-estimating relationships. The contractor's use of valid metrics or cost-estimating relationships is usual and customary. The evaluator must ensure that a logical cause-and-effect relationship is established between the variables based on reliable data and preferably actual work performed by the contractor. The evaluator working with the DCAA, in most cases, should examine the methodology and the currency of the data, as well as the reliability of the data for the proposed effort. When evaluating metrics or estimating relationships, the evaluator should be aware of the use of any old metrics and its validity to the current proposal.
Conversely, the evaluator should be aware of the use of newly established metrics and the ability to verify the data for the program. In addition, there are adjusted metrics that need to be defined and determined reasonable for the task, as well as tasks bid separately, where actual work performed by the contractor that produce the metric may have been included in the task. The bottom line for the evaluator in cost-estimating relationships--there is a clear need to eliminate any exposure to added hours.
Manpower analysis. When evaluating a manpower analysis for an effort over time, there should be a simple test of reasonableness because sometimes the sum of the parts is greater than the whole should be for the task. An example is XYZ program with a 24-month period of performance. The proposal is as follows:
Basis of estimate (BOE): 182,500 hours proposed over 150 hours per month for 24 months.
Rationale: The manpower analysis is 1,217 man months over 24 months.
Question for the evaluator: Does this equal the estimate of 51 equivalent full-time people for the life of the program? The evaluator should consider "Can I really envision 51 people working full time on this and only this program for 24 months?"
Manpower analysis, as with other estimates, uses statistics to establish reasonableness for the proposed tasks. The bottom line is to ensure the methodology and the statistical analysis is reasonable for the proposed task.
Skill mix. During the initial evaluation of the proposal, the evaluator assesses the skill mix for the labor hours. The evaluator must be confident that the skill mix is appropriate for the tasks proposed. Often, a contractor may bid higher skill levels than needed to do the job, and then not actually use these senior people during contract performance. The evaluator should ask these questions during fact-finding to help him/her find a fair skills mix for the proposed effort.
Man-loading estimates. These estimates are very common in most large corporation's proposals and are usually poorly supported. Man-loading estimates are those hours used for "monitoring," "coordinating," "interfacing," "supporting," or perhaps "problem-solving." These are the tasks that are half-time or indirect estimate of labor hours using statistics for justification. The evaluator must encourage the contractor to be more specific for the government to accept such estimates.
Judgmental estimates. These estimates are based on the experience of the estimator. There are professional judgments, engineering judgments, judgments based on experience, or other such ambiguous adjustments. The evaluator is searching for an adequate basis for the estimate. The contractor should fully explain how the judgment resulted in the hours proposed and the underlying assumptions inherent in the estimate, quantified with specifics. No matter who offers the judgment (government analyst, evaluator, or contracting officer), there must be some logical explanation for establishing a position.
Unsupported estimates. The contractor must support all estimates with specific rationale for the hours to be valid. The technical evaluation should only substantially reduce a task if it is not a valid task (with the agreement of the contractor). All decisions to eliminate or reduce task should be agreed to by both parties during fact-finding and not negotiations. An honest and open dialogue is needed to understand the contractor's position and discover the correct estimate for the proposed effort. The technical evaluator is accountable to ensure that the government's position is adequately communicated to the contractor and reasonable steps have been taken to reduce any perceived inconsistencies in the proposal.
Negotiations may be part of any major acquisition action, but they are a mandatory feature of a sole-source environment. The relationship of the buyer and the seller is usually adversarial, with the contractor defending the proposal estimate that is being technically evaluated. But remember--questioning the contractor's proposal should be done during the fact-finding and not the negotiation stage, so the conclusions of the technical evaluation are not surprising to the contractor.
To establish a solid negotiation position, the technical evaluator should review and assess the labor hours, the kinds and quantities of material, travel input, and other direct costs. DCAA and DCMA are responsible for the verification of labor rates, overhead rates, and material costs in an audit report and price analysis report. Together, the government and contractor teams establish a fair and reasonable position to meet the needs of the project and the contractor's return on investment.
A well-written evaluation defining a contract's requirement from cradle to grave will become a cornerstone and key document in fair and reasonable negotiation sessions. The key elements to the technical analysis are: (a) understanding the contractor's proposal, (b) organization, by reviewing the SOW and scope of tasks for sufficiency, (c) preparation, by reviewing the contractor's estimating methodology and rationale, (d) communication, through continuous fact-finding to determine fair and reasonableness of the contractor's estimates, (e) focus, writing to support party negotiations, and (f) consolidation of all elements, by evaluating the prices in the contractor's proposal and ensuring there are no disconnects or inefficiencies.
The lack of experience in program office personnel selected to perform technical evaluations is often a problem. In many programs, personnel with less than two years experience may be asked to evaluate medium- to high-complexity satellite and aircraft procurements with little direct guidance. Programs must (1) give adequate training to junior members and any potential technical evaluator, and (2) support the development of software tools to assist in the evaluation process. To resolve this problem, the government is moving toward developing training aids, checklists, and tools needed to support the technical evaluator offering more oversight and accountability for expending taxpayer's dollars.
The acquisition community must refocus its efforts and direction on ensuring a synchronized, well-documented, comprehensive, complete and accurate technical evaluation for sole source procurements. The basic facts presented here hopefully give a solid foundation and starting point for a technical engineer to go forth and accomplish an evaluation sufficient for negotiations.
Government Technical Evaluation Work Sheets Address each proposal element in an order that closely follows the contractor's proposal, to enhance comparability. In addressing direct labor hours by labor category, man-loading and quantity should be emphasized in the evaluation. Emphasize quantity and types of materials and other direct costs. Under travel, the number of man/trios, length of stay, and destination should be emphasized. Proposal Contractor's Air Force Air Force Air Force Elements Proposal Minimum (Low) Objective (High) Objective Notes (a) (b) (c) (d) (e) (a) Examples of common proposal elements include: (1) Direct lab hours (by labor category and fiscal year) (2) Material (3) Travel (4) Other/direct cost (b) Include in this column the quantity proposed rather than dollar value. (c) Include in this column the least quantity that the technical evaluator feels the contractor would require to accomplish the effort, and one that is adequate for an opening position. (d) Include in this column the maximum quantity that the technical evaluator feels the contractor could require to accomplish the effort (may be the same as (b) if reasonable). The Air Force objective amount (high) will be used for requesting negotiation approval. (e) This column shall contain an alpha reference to the rationale write-up for each exception taken to the contractor's proposal. (The referenced write-up shall include detailed remarks describing the analysis techniques and rationale used by the technical evaluator.) (1) Figure 1. Technical Evaluation Brand X Space Systems Div. Proposal XX-5542, Box 36A Security Upgrade Support 1. Introduction: This proposal is in response to RFP A293-001, dated 10 February 93. The proposal provides for technical and drawing support of the contractor during the installation of physical and visual security hardware. 2. A complete evaluation of the proposal has been made and is documented in the following format: (a) background, (b) approach, (c) evaluation of direct labor hours, and (d) proposal summary. Task 2 Project Coordination: This task provides Brand X program office support to this program office over the entire effort. The proposed 279 hours assumes a flat rate of .2 head for 9 months at 155 hours a month. The evaluation is task-based on the following estimates: * Estimate 9 monthly program reviews at 4 hours + 2 hours preparation = 46 hours * Estimate 12 weekly telecons during construction at 2 hours + 2 hours coordination and preparation = 48 hours * Estimate action item close out at 3 hours per week = 108 hours * Estimate Brand X program office support of 2 Brand X interim reviews at 4 hours = 8 hours Total program office support = 210 hours (46+48+108+8). Note that this program recently changed from a monthly to a quarterly review. A minimum position would reflect a project status once a quarter (i.e., 18 hours versus 46 hours). Recommend a minimum position of 192 hours and a target of 210 hours." (2) Figure 2.
1. SMC/CI Operation Instruction 64-1, Technical Evaluations of Contractor Proposals, Attachment 2, March 1999.
2. Technical Evaluations, presented by SMC/PKOM, sanitized for release.
Air Force Space Command Technical/Functional Evaluation Checklist.
"Fact-finding and Technical Evaluations," Aeronautical Systems Center Pamphlet 63-1 (September 30, 2000).
GPS Technical Evaluation Training, 2003.
"Project Officer's Guide for Preparing Technical Evaluations," SMC/MTK (August 24, 1999).
"Technical Evaluations," Electronic Systems Command, PKXF Briefing.
"Technical Evaluation of Contractor Proposals," CI Operating Instruction 64-1.
About the Author
ROBERT GRAHAM, CPCM, is currently deputy for the Space and Missile Center's Acquisition Center of Excellence. He has published numerous writings on the contracting profession and is Level III-certified in contracting. Send comments on this article to firstname.lastname@example.org.
|Printer friendly Cite/link Email Feedback|
|Date:||Apr 1, 2005|
|Previous Article:||World Congress 2005.|
|Next Article:||Governance meetings schedule.|