Competitive prototyping: a PMO perspective.
While most acquisition professionals feel, fundamentally, that "competition" is important, what are the positives and challenges of adding competition as part of a competitive prototyping process in support of an acquisition strategy? To answer this question, we will look at the competition process from a sitting program manager's (PM's) perspective and discuss the aspects and impacts of a competitive prototyping process during a program's Technology Development (TD) phase.
Our specific example is the Joint and Allied Threat Awareness System (JATAS-AN/AAR-59) program within Program Management Aviation-272 (PMA-272) (Advanced Tactical Aircraft Protection Systems), part of the Naval Air Systems Command (NAVAIR) Program Executive Office for Tactical Aircraft (PEO-T) at Naval Air Station Patuxent River, Md. The AN/AAR-59 is an Acquisition Category (ACAT) IC program that recently completed a competitive prototyping TD phase between two contractors. The competitive prototyping process resulted in a Milestone B decision and subsequent Engineering and Manufacturing Development (EMD) acquisition phase contract being awarded in early third-quarter Fiscal Year 2011. The AN/AAR-59's Initial Operating Capability (IOC) date is 2015 onboard the MV-22 Osprey aircraft platform.'
In the Beginning
The Counter/Counter Air Defense Initial Capabilities Document (ICD) dated June 15, 2006, established the need to increase the survivability of assault aircraft operating in hostile environments. In Fiscal Year 2007, the Chief of Naval Operations Air Warfare Division (OPNAV) N98 (Air Warfare Division) executed an independent Analysis of Alternatives (AoA) to evaluate alternatives to meet the capability gap identified in the Counter/Counter Air Defense ICD. The AoA results determined that threat-warning technology was mature enough to proceed with an advanced Missile Warning System (MWS). Subsequently, OPNAV/N98 drafted a Capabilities Development Document that designated the system as the AN/ AAR-59. The Department of the Navy approved the draft document to enter the Joint Requirements Oversight Council review process.
The JATAS team spent the summer of 2007 preparing program documentation for an intended program initiation at Milestone B. In September and November 2007, the Under Secretary of Defense for Acquisition, Technology and Logistics (USD[AT&L]) and the Assistant Secretary of the Navy for Research, Development and Acquisition (ASN [RDA]) released memos on prototyping and competition. Throughout the spring of 2008, the program worked with PEO-T, ASN(RDA), and Deputy Under Secretary of Defense for Portfolio Systems Acquisition to design the architecture of a competitive prototyping Technology Development Strategy (TDS). By the time the updated DoDI 5000.02 was released in late 2008, including a requirement for competitive prototyping, the program was already well under way with documentation and contracting plans for a competitive TD phase. The JATAS TDS was signed in December 2008, and in January 2009 the final Request for Proposal (RFP) for a competitive TO phase was released. In April 2009, then USD(AT&L) John J. Young Jr., concerned about the possibility of uncoordinated missile warning and countermeasure approaches on similar systems by each of the Services, issued an Aircraft Survivability Equipment (ASE) Acquisition Decision Memorandum (ADM). This memorandum directed a more coordinated solution for missile warning and countermeasures. The PMA-272 JATAS program was designated an Acquisition Category 1 (ACAT IC) special interest program, and the ASN (RDA) was designated as the Milestone Decision Authority. The ADM effectively endorsed the Navy Program Management Office use of competitive prototyping as part of its acquisition strategy.
TD Phase: the Heart of Competitive Prototyping
The JATAS program's acquisition strategy determined that a 16-month competitive prototyping TD phase would be used that included two prime contractors: Alliant Techsystems and Lockheed Martin. The effort would be managed via a costplus, incentive-fee contract. Each prime contractor completed a System Requirements Review, a System Functional Review, and a Preliminary Design Review that resulted in an approved allocated baseline for its respective AN/AAR-59 design. In addition, both contractors completed prototype ground and flight tests, and modeling and simulation were used to predict system performance.
JATAS's implementation of a competitive prototyping phase resulted in some intended and unintended consequences. To capture the positives and challenges that PMA-272 experienced as an outcome of implementing a competitive prototyping phase, various program stakeholders were interviewed and program documentation analyzed to gain insight into the PMA-272 competitive prototyping phase. This resulted in a white paper in November 2010 that identified, from the government's perspective, positives and challenges with respect to TD competitive prototyping during the AN/AAR-59 program. A synopsis of these findings follows starting with the positives.
* Responsive Contractors: Though the program was not technically in a source selection environment for a TD contract execution, the arrangement resulted in a "competitive environment" additionally fostered by the government team. Both TD contractors were extremely responsive to the government during TD contract execution, allowing the program to maintain its aggressive schedule. Both contractors were reluctant to exceed planned costs, even on cost-plus, incentive-fee contracts, because of the perception of competition. The performance of each contractor in cost, schedule, and performance was to be an important discriminator in the next phase of the competition.
* Competitive Future Pricing: As a result of a competitive prototyping effort, the program was able to award a Fixed-Price Incentive Firm (FPI F) contract for the EMD phase, an FPIF option for Low-Rate Initial Production, and Firm-Fixed Price (FFP) options for the first seven full-rate production lots, as well as FFP options to purchase hardware and software data rights to enable future competition. Prices were formulated by contractors in a competitive environment, resulting in the lowest possible cost to the government. Competing for production contract awards and purchasing the data rights are not typically affordable during a TD program.
* Technical Risk Reduction: Prototyping during the TD phase reduced technical risk for the AN/AAR-59 program. Both contractors, as a result of competition, made significant efforts toward early integration of their designs. Early looks at hardware before Preliminary Design Reviews (and analysis based on data collected during government prototype testing) increased government confidence in contractor assertions of predicted performance of the EMD designs. Prototype data from two separate approaches also allowed the AN/AAR-59 program to more accurately evaluate Technology Readiness Level. With all system performance being equal, the EMD and production contracts were able to be awarded based on total cost.
* Program Execution Risk Reduction: Dual contract execution in a competitive environment allowed the government team to observe in real time the effectiveness of the corporate management systems, earned value performance, program management, and contract execution of the TD contractors. The government team was able to leverage this experience and insight to assess program execution risk for each contractor during the EMD source selection. Personal relationships were developed between the contractor and government teams, reducing the time required during EMD for the joint team to become effective. As a result, the teams developed a clearer understanding of areas that led to effective communication paths, roles, and responsibilities.
Additionally, both the AN/AAR-59 system and the government's technical team matured during the TD phase. The technical team had a chance to observe two technical approaches, exposing team members to greater technical insight compared with monitoring a single development phase. As a result, the government team members felt they would be more effective during EMD because of this experience.
Also, a set of documentation core to the program was developed during the TD phase. These core program documents then could be leveraged into the development of other products such as the Acquisition Strategy, Systems Engineering Plan and the Test and Evaluation Master Plan supporting the EMD phase. Finally, there was the opportunity to collect sensor data to support development of new algorithms. These data would have been required for completion of EMD, but waiting to collect them during EMD would have introduced additional technical and schedule risk much later into the program's acquisition life cycle.
While there were positives associated with PMA-272's JATAS competitive prototyping efforts during the TD phase, there also were some program challenges.
* Government Workload: Administering two TD contracts increased the government workload without a comparable increase in team size, doubling the meetings, Contract Data Requirements Lists to review, and contract administration. The AN/AAR-59 government team executed eight major reviews (two each of Integrated Baseline Reviews, System Requirement Reviews, System Functional Reviews, and Preliminary Design Reviews--instead of one each) in 13 months, while simultaneously preparing for EMD source selection and a Milestone B review. In addition, the Systems Engineering Technical Review events doubled the attendance requirements for senior engineering and logistics competency members to sit as board members and provide subject matter expertise.
A case could be made for the necessity of three teams successfully executing a competitive prototyping effort. To prevent the government from inadvertently leveling the technical solution by having the same government team deal with both contractors, the AN/AAR-59 Logistics and Engineering disciplines built two teams to interface directly with each of the two vendors. The need for a third team immediately was evident to provide the link between the two, prepare for the milestone decision, and build the RFP for the follow-on for the EMD acquisition phase. The net effect of competitive prototyping was a large increase in workload for the JATAS Program Management Office team.
* Reduced Government Influence on JATAS Design: As a byproduct of the competitive environment in the TD phase, the government played a limited role in influencing the JATAS engineering design. To avoid "technical leveling" between the two TD contractors, the government team restricted itself to (1) ensuring the contractors fully understood the government's requirements and intent, (2) ensuring the government fully understood each contractor's approach to meeting the requirements, and (3) providing guidance about perceived risk if a particular approach might fall short of government requirements. Beyond that, the government explicitly did not provide specific technical direction or design solutions to the TD contractors. As a result, the government's ability to influence the JATAS design early in its development was limited. Similarly, the government was limited in its ability to adopt good concepts from either TD contractor into its requirements because the JATAS specification could not be changed for the benefit of one contractor over the other.
* Timing Issues With Gate Review Process/Contract Gap. The Secretary of the Navy (SecNav) Gate Review Process (see Figure 1 on p. 21) did not optimally align with the TD competitive prototyping strategy as it did not allow execution of the EMD source selection in parallel with execution of the TD contracts. The result was a gap between the end of the TD period of performance and award of the EMD contract because the RFP for the EMD source selection could not be released until Preliminary Design Reviews were complete and the Capabilities Development Document was signed, essentially at the end of the TD phase. The impact of this "gap" was an additional expenditure of funds to extend the TD contracts and, as a result, the program realized a schedule delay in starting the EMD phase.
* Workload Issues with DoDI 5000.02 and SEC-NAVINST 4105.1B and Certification Requirements. The SecNav Independent Logistics Assessment process required a review of supportability plans prior to a single vendor down-select at Milestone B. This created a situation in which documentation from vendors could not be provided to assessors without nondisclosure agreements, and, in some cases, special training due to the nature of source selection sensitive materials. This resulted in an overarching Life Cycle Sustainment Plan being evaluated and not the initial product support strategies from the vendors, in effect tripling the workload of the logistics team in this area.
In addition, the timing of the Independent Logistics Assessment was too late in the process to be truly effective. To restore a proactive stance to the assessment, it should be conducted prior to releasing the RFP instead of a prescribed length of time prior to the milestone. Focusing only on the milestone resulted in passing the window of opportunity to influence the Statement of Work, Contract Data Requirements Lists, and all other associated deliverables (and their timing).
After looking at the positives and challenges associated with PMA-272's competitive prototyping during the TD phase, several lessons were learned from this experience.
* The program did not have an established Acquisition Program Baseline when the competitive prototyping policy guidance was released. Upfront and early communication with Navy and Office of the Secretary of Defense staff identified that this emerging policy would be applicable to this program. Close collaboration with the resource sponsor and policy authorities allowed the program to define a TD strategy intended to meet the needs of all stakeholders. Staying abreast, and, in some cases, ahead of emerging policy guidance can help a program team make progress in a changing environment.
* The standard process requirements of maturing a system through Preliminary Design Review will limit the number of contractor teams that can be effectively managed during competitive prototyping.
* The complexity and number of the system interfaces (internal and external) required to successfully field a system will limit the number of contractor teams that can be effectively managed during competitive prototyping.
* Competitive prototyping requires the government team to practice thoughtful engagement when dealing with the contractor team. The government team member must stop and think of the competitive environment before directing or responding to the contractor. This requires the government team member to look at all decisions from multiple viewpoints. Hopefully, this will result in a more thoughtful response.
* The JATAS team felt that a single government team working to ensure its direction would not result in technical leveling, but would provide better and more balanced leadership than two government teams with strict firewalls.
* It is important to make sure the team agrees on how the competition will be conducted. The contracts and technical teams may have different views regarding how best to maintain a "level playing field." To the program manager, procuring contracting officer, and technical team, playing "fair" may consist of providing both teams the same information.
* When the program is to down-select from competitors, plan for the gap that will fall between the final demonstration/presentation and award of the follow-on contract. JATAS chose to have both teams continue on risk-reduction efforts.
* There is a price for competitive programming, and the program will need sufficient funding.
* The maturity of the system specification going into a competitive prototyping environment is extremely important as there potentially will be at least twice the number of requests for clarification. This increased workload will only further stretch already limited program resources.
* The user community must be involved early and invested in helping create the system the warfighters need. The platform office provides the detailed information regarding the environment in which the system will be operated and maintained once it has been installed successfully.
* Logistics and Test and Evaluation are part of the technical team and should be invited to the engineering meetings whenever possible.
* When a program uses competitive prototyping during the TD phase and the EMD contract is to be Fixed-Price Incentive-Firm, it can be difficult for the government team to have a meaningful dialogue with the contractor and provide technical direction and insight without causing "technical leveling" during TD or out-of-scope requirements during EMD.
Though the final chapter is yet to be written, the AN/AAR-59 competitive prototyping effort appears to have been a technical success. But the savings and/or costs associated with resources and schedule are still unknown. High-risk TD programs should consider this prototyping strategy as a risk mitigation strategy. In addition, the competitive nature of the contracts forces the contractors to be responsive to cost, schedule, and performance during the system's development.
It is hoped that the sharing of PMA-272's competitive prototyping positives and challenges may help future program management teams as they make a determination to include competitive prototyping in their acquisition strategy, and that the lessons learned can add to the proficiency of the process.
The authors can be contacted at email@example.com, Duane.Mallicoat@dau.mil, and Bradley.Bates@dau.mil
Capt. Paul Overstreet, USN * Bradley Bates * Duane Mallicoat
Overstreet is the F-35 Weapon System program manager and the former PMA-272 program manager, Bates is a DAU Mid-Atlantic professor of acquisition/program management, and Mallicoat is the DAU Mid-Atlantic Region associate dean for Outreach and Mission Assistance.
|Printer friendly Cite/link Email Feedback|
|Author:||Overstreet, Paul; Bates, Bradley; Mallicoat, Duane|
|Publication:||Defense AT & L|
|Date:||Mar 1, 2013|
|Previous Article:||Defense acquisition workforce awards for 2012.|
|Next Article:||Big 'a' systems architecture: from strategy to design: systems architecting in DoD.|