Revitalizing systems engineering: how six components are meeting the acting USD (AT & L) imperatives.

In the March-April 2005 issue of Defense AT & L (pages 14-17), Michael W. Wynne, acting under secretary of defense for acquisition, technology and logistics, and Mark D. Schaeffer, principal deputy, defense systems and director, systems engineering, Office of the USD(AT & L), called for the revitalization of systems engineering across the Department of Defense. "Analyses of a sampling of major acquisition programs show a definite linkage between escalating costs and the ineffective application of systems engineering," Wynne and Schaeffer wrote.

[ILLUSTRATION OMITTED]

In a February 2004 policy memorandum, Wynne issued a directive to meet the problem: "All programs responding to a capabilities or requirements document, regardless of acquisition category, shall apply a robust systems engineering approach that balances total system performance and total ownership costs within the family-of-systems, system-of-systems context."

Wynne and Schaeffer called for "systemic, effective use of systems engineering as a key acquisition management planning and oversight tool" and said that in addition, DoD would "promote systems engineering training and best practices among our acquisition professionals."

Defense AT & L presents the responses of six defense components to the acting under secretary's call to arms.

Systems Engineering Outreach: A DCMA Perspective

The Defense Contract Management Agency's systems engineering revitalization efforts include creating a customer-focused, performance-based organization that encourages creativity and uses customer-driven measures; and providing product assurance services that meet or exceed customer expectations. DCMA has embarked on a redefinition of its traditional quality assurance services to an acquisition life cycle-based comprehensive product assurance program. Increased participation using systems engineering processes and practices during the system development and demonstration phase is seen as key to implementing this new approach. Engineering and critical thinking are increasingly important in these revitalization efforts and resource management, skills management, and supplier management (which include working with industry to improve supply chain management as well as benchmarking with other organizations).

The application of systems engineering processes and practices enables the DCMA engineers to correlate risks with contract performance requirements. The DCMA analyses result in a prediction of the impact on performance, cost, and schedule, which allows for early corrective action. This, coupled with recommendations and opinions, supports the program manager's goal of providing a successful weapon system program within cost and schedule constraints.

Integrating Risk Management

Based upon programmatic outcomes, DCMA has instituted an integrated system of risk management to provide acquisition program managers and their organizations with focused acquisition support. Increased focus on high-risk events identified throughout the product life cycle is fundamental to this new approach. Key components of the system are the in-plant surveillance for engineering, software development, quality assurance, and manufacturing processes. DCMA personnel analyze trends of key performance parameters by using technical performance measures against planned baselines in assessing impact on acquisition milestones. Cost and schedule impacts are assessed on the basis of the in-plant surveillance, with inherent projections of future cost growth and schedule delays predicated on the attainment of key performance parameters.

DCMA strives to drive consistent engagement in the system development and demonstration (SDD) phase by providing a framework for engaging with its customer base to account for unique activities in the SDD environment; the definition of roles and responsibilities; and the development of consistent assessment tools, techniques, and metrics for the entire life cycle. Additionally, the introduction of an interdisciplinary teaming approach is viewed as essential. It will assure that suppliers' plans and processes are capable of meeting customer outcomes and are effectively executed; and that the process interfaces that drive product quality are identified and operating effectively. Early interface with the customer through customer-outcome strategy meetings are to be used to identify and clarify customer outcomes and performance measures early in the program. The information extracted is used to develop unique program-based surveillance strategies that provide for the early identification and analysis of program risk factors, critical product characteristics and processes, and risk-consequence information. Also, increased effort in the SDD phase using systems engineering methodologies will ensure that a proper foundation for program execution is established, that risk assessment and mitigation are addressed, and that potential program impacts are forecast early.

Capability Maturity Model Piloted

DCMA is piloting the use of capability maturity model integration (CMMI) as a tool to determine the risk associated with suppliers' systems engineering processes. CMMI-based risk management methodology is targeted for ACAT I and II programs in the technology development or system development and demonstration phases. This method helps DCMA engineers to identify and prioritize the most critical supplier processes; to evaluate those processes objectively relative to industry's best practices as defined in the CMMI; to identify suppliers' process strengths and weaknesses and the impact on product and program performance; and to assess program and product risk--along with other measures, such as earned value and technical performance measurements--and predict future program outcomes.

DCMA is also continuing efforts to improve the skill levels of its engineering workforce by developing internal courses, improving the guidebooks, and participation in the INCOSE [International Council On Systems Engineering] Systems Engineering Certification and Sabbatical programs. The latter allows DCMA employees to continue full-time studies at a local college or university for a period of 18 months. Booz Allen Hamilton, under contract with DCMA, has developed a list of the general and technical competencies needed for the engineering workforce, and efforts are under way to develop career guides.

R. Pillai, deputy director. Contract Technical Operations

Coupling Acquisition and Systems Engineering Processes at DISA 

The Defense Information Systems Agency is acquiring more complex systems to fulfill the mission of providing global, net-centric solutions to warfighters. DISA is challenged to improve the time to market of these products and services (network-centric enterprise services and Internet protocol convergence, for example), while ensuring they satisfy users' needs. We believe that a close coupling of acquisition and systems engineering processes throughout the life cycle is essential.

DISA's recent transformation initiatives have enabled the agency to progress toward an integrated acquisition/systems engineering environment as--quite simply--the way we do business. Six key components of DISA's efforts follow.

Demonstrating Senior Leaders' Support

It starts at the top. As shown repeatedly in industry best practices, the foundation for institutionalizing systems engineering is the commitment of the senior leaders, demonstrated through action and communicated throughout the enterprise to instill staff commitment. DISA leadership has done just that over the past 18 months, and two key actions are particularly noteworthy.

In October 2003, DISA created the component acquisition executive (CAE) office in accordance with DoDD 5000.2 to implement DoD acquisition policy and guidance and to oversee and guide the acquisition of all programs or projects. The CAE reports directly to the DISA director, is the line of authority for all program managers, and is responsible for representing the agency within the broader OSD-level acquisition community.

DISA also created a systems engineering organization that works in coordination with the CAE office to "plan, engineer, acquire, and integrate joint, interoperable, secure global net-centric solutions satisfying the needs of the warfighter and develop and maintain a first-class engineering workforce to support the needs of DISA's programs."

The DISA director identified world-class acquisition and world-class engineering as two of his top 10 transformation initiatives for the agency. This vision and the new organizational constructs serve as the foundation for institutionalizing systems engineering rigor for all DISA programs, regardless of the acquisition category.

Documenting Processes

The first step was to agree upon and document a set of repeatable systems engineering processes. We formed a working group of engineering leaders from across the agency to oversee the effort. The resultant DISA systems engineering process document incorporates best practices from DoD, industry, and academia, coupled with many decades of systems engineering experience represented within the working group. It addresses:

* Activities, milestone events, and products to be accomplished throughout the acquisition/engineering life cycle

* Mandatory systems engineering artifacts (e.g., project schedule, systems engineering plan, joint capabilities integration, and development system products)

* Entrance and exit criteria for key reviews (e.g., technical requirements review)

* Guidelines for tailoring the systems engineering processes

* Cross-program engineering processes to address critical program interdependencies for DoD's future net-centric environment.

Additionally, the systems engineering working group established ongoing process improvement mechanisms, enabling DISA's documented processes to evolve as new programmatic, technological, or operational challenges arise.

Training the Professional Workforce

We needed to provide our acquisition managers and systems engineers with the right level of training to implement systems engineering with rigor within their individual programs/projects. To this end, we have reinvigorated the agency's engineering career management program and continue to increase the number of certified engineers in our workforce. We have developed a partnership with the Defense Acquisition University to integrate DISA's systems engineering processes, software, and network engineering best practices and net-centricity tenets into the DAU training curriculum.

Governing Systems Engineering Implementation

We adopted a phased approach to implement the systems engineering processes across DISA's programs/projects. It began with a three-month pilot period, where our primary goal was to validate the documented processes and create a repository of systems engineering products. In subsequent phases, an increasing number of programs have been earmarked for inclusion until all programs/projects have adopted the DISA standard processes.

DISA's ongoing governance structure involves ensuring compliance with documented processes and ensuring that engineering content is sound and meets stated and implicit requirements. We have initiated multi-tiered reviews and associated governance organizational structures to assess systems engineering implementation within a program and across interrelated DISA programs, and to verify compliance in both areas from an end-to-end global information grid perspective. The CAE has already established joint program reviews, supported by the systems engineering organization, that address both acquisition-focused and engineering-focused topics. Quick-look technical assessments are being conducted for each program/project to identify best practices and recommended areas for improvement. Event-driven peer reviews are being initiated to focus on areas where it is deemed that further review is necessary.

Sharing Best Practices and Lessons Learned

We require all mandatory systems engineering artifacts to be posted on the DISA intranet so they are accessible across the agency. This practice improves efficiency by allowing managers and engineers to review and reuse relevant data and methodologies. It can also lower risks by ensuring cross-program consistency and preventing repetition of past mistakes.

Measuring Success

No process is complete without tracking progress and measuring the extent to which objectives have been achieved. We have defined systems engineering metrics in alignment with DISA's agency-wide balanced scorecard initiatives. Systems engineering-specific balanced scorecard metrics address such areas as posting mandatory artifacts, sharing best practices, conducting independent technical assessments, and meeting schedules. We require these and other related metrics to be reported to management on a quarterly basis.

Rebecca Cowen-Hirsch, deputy component acquisition executive; Rebecca Harris, principal director, Global Information Grid Enterprise Services Engineering; and Dave Mihelcic, chief technology officer

Applying Systems Engineering to IT at the DLA 

Since 1999, the Defense Logistics Agency has been dedicated to a transformational effort to re-engineer its business practices in response to changing warfighter logistics needs. Currently implementing a number of commercial off-the-shelf (COTS) programs, including a major enterprise resource planning system, DLA is at the forefront of business systems acquisition and is creatively applying systems engineering approaches to information technology (IT) programs through a tailored business systems engineering approach.

Though--as with any major acquisition program--the foundation for DLA's approach to business systems engineering is the defense acquisition system, the agency has developed a structured and repeatable business systems engineering process in reaction to several DoD- and congressionally mandated initiatives to improve the management and delivery of information technology programs. Guiding this process from desired capabilities to IT business solution implementation, DLA relies on best industry COTS solution-integration practices, which are now being embedded, along with other best practices, in the Logistics Domain-sponsored "Enterprise Integration Toolkit," which can be found at <www.eitoolkit.com>. As DLA has discovered through the application of these business systems engineering principles within its business systems modernization program. COTS-based IT programs must rely on a disciplined but timely life-cycle process that maximizes best business practices and the lessons learned from large-scale COTS-based implementations in industry.

Leveraging Industry Best Practices

The following examples illustrate a few of the best industry practices used by DLA as it applies business systems engineering principles to introduce its new COTS-based business systems environment.

DLA partners with a leading practitioner from industry, whose technical expertise and integration experiences reduce implementation risk and provide the capability to maintain pace with the rapid changes in COTS technology. A critical element of the business systems engineering approach is the requirement for technical reviews directed at data integrity and data quality.

DLA ensures compliance with the enterprise architecture and supporting architecture artifacts. The systems, technical, and operational architecture views take on an extremely important role in the design and implementation of business systems. This, combined with functional requirements traceability, ensures that the technical solution can provide required capabilities in a "to-be" environment at every level of the enterprise. The revitalization of systems engineering with the focus on the technical management of business systems will contribute to defining the net centricity, interoperability, and business enterprise architecture compliance processes and criteria that are currently evolving.

DLA employs incremental and spiral development approaches within the business systems implementation environment. In many cases, the full functionality of the COTS-based system is implemented with the first release but deployed to a limited number of users or a manageable segment of the business. Future increments do not add functionality but are directed at increasing the number of users or business volume. This results in the identification of needed improvements or enhancements to ensure compatibility with the business environment. The identification of functional requirements may be defined in phases as technology matures or as the benefits of technology are better understood. This leads the program towards a spiral development approach.

We make optimal use of testing and evaluation processes to reflect the inherent differences in a COTS-based system environment. COTS testing is focused on operational assessments, followed by formal initial operational test and evaluation once operationally ready. The test strategy is tailored to fit the risk and complexity associated with the business systems solution and incremental and spiral development approaches.

A systems engineering plan is key to the revitalization of systems engineering in the business systems environment. The documented technical management approach within the business systems engineering process addresses the risks and concerns surrounding business systems programs and provides the tailored approach to effectively manage, design, test, and deploy critical business systems solutions.

Continued emphasis on institutionalizing these business systems engineering principles is a DLA priority as the agency acquires and introduces more and more COTS-based business systems. These principles, as well as the other best practices embedded in the EI Toolkit, continue to blend DoD-unique best practices with the best business practices of industry. The result will be a continuing enhancement of defense acquisition system processes to reflect the unique characteristics of IT and business system acquisition.

David J. Falvey, program executive officer, information operations 

Reinvesting in Systems Engineering in the Department of the Navy

Since the end of the Cold War, more than 75 specialized defense firms and/or divisions have merged into five major contractors. Consequences of this consolidation were the breakup and realignment of experienced engineering teams and processes, and the loss of systems engineering expertise as a result of retirements and downsizing. Meanwhile, the government downsized functions viewed as ancillary (that is, considered as overhead) to the Services' mission of winning wars. Therefore, revitalization and reinvestment in systems engineering are necessary prerequisites for the challenge of specifying, designing, and fielding the systems that must operate in the networked family-of-systems/system-of-systems (FoS/SoS) environment of the transformed forces of the future. This will include introducing new processes and tools that scale up to globally distributed systems and identifying the people needed to implement and lead systems engineering efforts in both government and industry. That revitalization is under way across Navy and Marine Corps programs at three levels: traditional systems developed by program managers; Navy and Marine Corps FoS/SoS programs that are not under the purview of a single program executive office: and at the international partner coalition level. This section of the article addresses the first two.

Revitalization at the Core Program Level

The Department of the Navy (DoN) acquisition community has organized around the engineering challenges and design practices that are unique to ships, submarines, aircraft, and land units, and to the command, control, communications, computers and intelligence (C4I) infrastructure that brings them together as a cohesive fighting force. Responsibility for gaining and retaining corporate knowledge, the technical expertise, and tailored systems engineering practices is assigned to seven systems commands and their associated program executive offices (PEOs) and program managers. In 2002, the concept was adopted of a virtual systems command that incorporates a systems engineering stakeholder group to efficiently integrate systems engineering processes between the SYSCOMs. The stakeholder group is pursuing a number of systems engineering revitalization initiatives, a few of which are described here.

Naval Systems Engineering Guidebook

A Naval systems engineering guidebook was developed to document a common systems engineering process. It leverages industry and government best practices and documents the critical systems engineering processes typically associated with acquisition programs.

Aviation Ship Interface Specification Guide

A joint-Service specification guide on air vehicle/ship integration is being developed. It will enable future ship-builders and aviation equipment suppliers to develop more robust and complete specifications, thereby aiding systems engineering by allowing more effective aircraft integration into ships.

Naval Systems Engineering Technical Review Process

A new systems engineering technical review process instruction has been issued to define processes and requirements for engineering reviews and to provide associated tools and instructions for consistent risk management.

Naval Technical Authority

A process and organizational framework was developed to designate individuals with the requisite subject matter expertise to certify that aircraft, ships, submarines, craft, and aircraft systems and weapons are safe to operate. This framework assigns authority, responsibility, and accountability, and it implements formal procedures to train, certify, and warrant individuals in defined technical domains to participate on PEO and program manager systems engineering teams.

Investing at the Family-of-Systems/System-of-Systems Level

The Naval capabilities evolution process (NCEP) has been created to apply the principles of systems engineering at the FoS/SoS level to transform from requirements-based to capability-based acquisition. The NCEP implements a mission-oriented, capability-based acquisition approach to engineer and field Navy and Marine Corps combat, weapon, and C4I systems that must operate as an FoS or SoS to deliver and evolve capability. Systems engineering integrated product teams are formed to derive, allocate, describe, and document system performance and interfaces among the FoS/SoS programs in a system performance document.

The NCEP includes three sub-processes--capability evolution planning, the capability engineering process, and the portfolio execution process--and key activities.

The capability evolution planning process supports the pre-Milestone A activities. It addresses the creation of acquisition portfolios for FoS/SoS systems engineering and for identifying the initial system functional and performance allocations, and the interface relationships among the portfolio systems. This process creates the capability evolution description of warfare system capability increments and fielding plans based on the planned evolutionary development of portfolio systems.

The capability engineering process supports the pre-Milestone B activities. Systems engineering principles are applied to perform detailed functional and performance analyses and design synthesis at the FoS/SoS level to refine performance allocations, and to identify key system interfaces and integration and interoperability requirements among portfolio systems. The product of the capability engineering process is the system performance document to be used by acquisition portfolio program managers for defining their programs.

The portfolio execution process also supports the post-Milestone B activities. It involves continuously monitoring the execution of acquisition portfolio programs to ensure that the desired capability is being evolved according to the capability evolution description, the system performance document, and the direction provided to individual programs. The portfolio execution process recommends courses of action to investment decision makers based on changes that occur to one or more portfolio programs.

Approval of each initial capabilities document or capability development document that affects an FoS/SoS-delivered capability should trigger an iterative pass through the NCEP. For those systems that support multiple missions, the NCEP activities will be performed for each mission or warfare system that is affected.

Carl R. Siel Jr., ASN (RD & A) chief engineer

Making Systems Engineering the Cornerstone at NGA 

Revitalization of systems engineering is the cornerstone of activities to improve the acquisition management capabilities at the National Geospatial-Intelligence Agency. As NGA continues to acquire more complex systems and services, the importance of having a world-class acquisition workforce is paramount. Since January 2000, NGA has been conducting activities focused on improving the proficiency of the acquisition workforce and the policies and practices they use. All efforts are showing success in improving acquisition agility and programs' success, and--most important--delivering systems, geospatial-intelligence, and services of higher quality to NGA's customers.

The basis has been defining new systems engineering and program management processes and improving existing processes within the Acquisition Directorate. Using the Federal Aviation Administration Integrated Capability Maturity Model as the reference model, 10 process areas were identified as critical to NGA's efficient and effective execution of acquisition management. Under the senior sponsorship of William Allder and Jaan Loger, former directors of acquisition, and led by the systems engineering process group, eight process working groups melded existing activities with industry best practices then documented and implemented repeatable processes to yield predictable positive results. Once implemented and institutionalized, the processes have yielded good results. Development and delivery schedules and customer satisfaction across the life-cycle activities improved, with fewer heroics.

While the original goal was to improve practices commensurate with Capability Maturity Level 2, a formal external appraisal conducted in October 2003 using a continuous representation model, found eight process areas at Level 3, one at Level 2, and one at Level 1. Process improvement proceeds with continued institutionalization, development of new processes, and implementation of some processes across the entire agency.

Systems engineering revitalization was expanded to include NGA's joint systems engineering work with one of its mission partners. Based on several NGA processes, joint systems engineering processes and a joint systems engineering management plan were collaboratively developed and implemented. These processes reflect the integrated and collaborative practices essential when working in a complex joint environment. Since November 2002, the use of these processes and joint systems engineering forums has reduced program risks.

Workforce Education: Critical to Revitalization

An important element in revitalization has been the acquisition management education program. To continuously improve the quality of the acquisition management workforce, in addition to Defense Acquisition University courses, NGA's Acquisition Management Professional Advisory Board and NGA senior leadership sponsor and fund several professional and personal development opportunities. The opportunities focus on improving the systems engineering, program management, and leadership competencies necessary for successful program execution. Partnering with The George Washington University and the University of Missouri-Rolla, NGA offers a two-year, on-site program towards a systems engineering graduate certificate to all civilian and military employees, contractor partners, and other government agency partners. Certificate graduates can go on to complete the final six classes in the NGA-sponsored master's degree program.

These on-site courses provide depth in particular areas of systems engineering and program management important in NGA's systems development, and they strengthen the students' discipline, efficiency, and effectiveness. Since February 2001, 76 students have received their systems engineering certificates, and the first master's program class of 18 graduated in April 2005. The seventh certificate cohort began in January 2005, and a second master's cohort is planned for the fall.

Developing Leadership

Strong leadership skills are also essential to developing successful systems engineering and acquisition management professionals overall. Based on a very successful leadership development program started in 2001 for NGA's contract management personnel, the Acquisition Leadership Development Program (ALDP) began in January 2004 for Acquisition Directorate systems engineers and program managers who demonstrated leadership potential. In December 2004, the first ALDP class of 25 students and third class of 21 contract managers graduated. ALDP 2005 began in January and incorporates improvements recommended by the first class and senior leadership.

To supplement ALDP and other leadership courses is a shadowing program in which junior personnel are paired with NGA and non-NGA senior leaders. Shadows spend one week observing and often participating in senior leadership activities.

NGA remains committed to improving the quality of its acquisitions, in part through the continuing improvements in the conduct of systems engineering practices and acquisition management overall. We will continue to place great emphasis on the professional development of the workforce and the means whereby members fulfill their individual missions and that of the agency.

Dr. Thomas H. Holzer, acquisition engineering technical executive

Institutionalizing Systems Engineering and Architecture Throughout Cryptologic Activities at the NSA/CSS 

On April 11, 2003, the National Security Agency director established the position of chief systems engineer, National Security Agency/Central Security Service (NSA/CSS), combining within it the unified cryptologic architecture and NSA/CSS systems engineering authorities. This position represented a milestone in institutionalizing systems engineering and architecture (SE/A) discipline and rigor throughout cryptologic activities and implementing the DoD systems engineering policies and directives.

Maturity on Multiple Fronts

There are five major aspects to the NSA/CSS systems engineering program: processes; architecture; SE/A analyses; integration analysis and support; and planning and resource (financial and personnel) management.

All major policies/directives are in place, with processes being implemented: deployment management; systems engineering; software engineering; configuration management; test and evaluation; modeling and simulation; and strategic enterprise management.

NSA/CSS systems engineering processes enable the broad-reaching, scalable implementation of systems engineering and decision support throughout the extended cryptologic enterprise. An overarching systems engineering policy is in place, providing the authority and responsibility for implementing SE/A. Additional policies enable implementation of configuration management, modeling and simulation, and deployment management processes. Configuration management and deployment management offer critical support to acquisition efforts, facilitating interface definition, integration, and gap analysis.

NSA/CSS systems engineering is also an integral component in numerous agency executive management activities and associated processes/policies. These include test and evaluation policy and document coordination; NSA/CSS strategic integration management process (providing information and objective assessments regarding capability gaps, analysis of alternatives, and cost estimation); and acquisition processes (providing systems engineering program-level support and documentation development/review).

NSA/CSS systems engineering is a transition partner with Carnegie Mellon University's Software Engineering Institute and its work on the capability maturity model integration. Internally, NSA/CSS systems engineering sponsors formal CMMI training and a broad systems engineering training program curriculum. Further, the agency has nurtured two employees through extensive training and hands-on performance, leading to their becoming authorized CMMI lead appraisers.

Unified Cryptologic Architecture

The unified cryptologic architecture and its constituent components represent a consistent organizing framework of information that provides enterprise context, constraints, and interface guidance (over time) to management, developers, and users.

Today's intelligence issues require extensive interoperability, data exchange, and collaboration. The complexities among the various Intelligence Community and DoD agencies create problems. The unified cryptologic architecture and NSA/CSS enterprise architecture combine established standards, an interface tree, a common service taxonomy, a data model, and DoD architectural format products to facilitate interoperability.

Broad Support Functions

NSA/CSS systems engineering provides direct support to the agency's acquisition programs. ACAT I programs have forward-deployed systems engineering personnel reporting to the program managers but also matrix-managed by the chief systems engineer. Thus NSA/CSS systems engineering policies, processes, and directives are institutionalized within major development efforts.

NSA/CSS systems engineering has a critical corporate-level role in addition to direct acquisition support, participating in the program planning and budget and execution process and identifying the need for new acquisition efforts, rather than simply supporting acquisitions already under way. NSA/CSS systems engineering reviews all program documentation and has signature authority on the systems engineering, information support, and test and evaluation management plans.

NSA/CSS systems engineering manages the evolution of the NSA/CSS cryptologic systems baseline, ensuring smooth integration of new capabilities into operations and adherence to the DoD online standards. Using a network of systems engineering personnel strategically placed within major programs and organizational elements, combined with key infrastructure/information management artifacts (such as corporate data repositories and an enterprise-integrated master schedule). NSA/CSS systems engineering performs integration planning and analysis across the breadth of acquisition activities.

Planning and Financial Management

The unified cryptologic architecture provides a common platform for resource planning, coordination, and alignment among the unified cryptologic system partners. The FY2006-2011 Cryptologic Planning and Programming Guidance is updated yearly and focuses the extended enterprise on key issues to facilitate interoperability and integration among the partners.

Within the agency, the NSA/CSS chief systems engineer participates in the corporate planning process. In addition to examining agency activities across the board to ensure cost and integration realities, the chief systems engineer also performs the planning and financial management for systems engineering activities throughout the agency's global enterprise.

NSA/CSS has made significant progress in capturing the collective corporate knowledge, documenting the future vision, and establishing policies/directives required to effectively system engineer the evolution of the cryptologic technical baseline.

Kelly A. Miller

Questions and comments for the authors should be addressed to [e-mail].