Making program assessment work: a profile of the U.S. Air Force Academy.

As THE DELECTABLE SCENT of Mom's apple pie permeates a kitchen, so should carefully constructed assessment affect every corner of an institution. At the United States Air Force Academy, each course or program has a course director responsible for evaluation; each department has an assessment director or coordinator responsible for all aspects of assessment at that level, and, of course, assessment is carefully orchestrated institution-wide by a number of individuals. In this article, we will examine all three levels, pointing in particular toward some common elements useful to all assessment efforts.

A look at one program

The whirring rattle of a diamondback poised to strike Maj. Brian Miller and a field as barren as Kevin Costner's Field of Dreams would seem like inauspicious beginnings for another dream: the vision of a Field Engineering and Readiness Laboratory (FERL) at the U. S. Air Force Academy. The vision of then-Colonel David Swint (now Brigadier General, Retired), FERL was conceived because he believed that students, particularly civil engineering (CE) students, learn by doing and then by thinking about what they are doing. He thus conceived of a place remote from classrooms, but still connected, where cadets could drive bulldozers, lay concrete, and construct wooden buildings. The premise behind his vision was, "Construct First; Design Later." The resulting three-week field experience at the end of the sophomore year is still unusual in engineering schools but has been a success. It is a documented success because, as Maj. Mark Malone (2003) put it, "Brig. Gen. Swint had the foresight to build in assessment from the get-go."

Buy-in. To make the vision a reality, the CE department began with a serious study of the curriculum. In weekly departmental meetings, the faculty reviewed twenty-five courses and "identified the fundamental engineering principles we wanted to focus on during the FERL experience. Then we developed cadet-built projects designed to demonstrate those principles" (Swint 1996, 4). The department also held day-long "assessment off-sites." The work within the department occurred "within a broader context of customer feedback and strategic guidance from the engineering division meetings, the dean of the faculty Policy Review Council, USAFA Board of Visitors and ... senior USAF leadership." Former faculty members were also consulted (Civil Engineering U. S. Air Force Academy Quality Air Force Unit Self Assessment, 11).

Thus, there was not only "grass-roots" buy-in from the department members, but the buy-in also included the entire U.S. Air Force because "we wanted to design a program that would introduce students not just to what they would see here, but what they would also see in the operational Air Force ... Dreams don't happen except by everyone buying in," concluded Brig. Gen. Swint (2003).

External reviewers. Besides the self-study and widespread buy-in, the CE department anticipated a recommendation by Kleniewski (2003, 2) for win-win program reviews: "A good review process makes wise use of external visitors." Many assessment experts became involved with the project--initially over twenty individuals, including active duty and reserve civil engineers--but the most helpful involvement came from the Air Force Armstrong Laboratory, which consented to direct an assessment effort in January 1994. Besides their own lab experts, they brought additional team members from the University of Georgia and the University of Colorado at Denver. The team recommended an advisory board and two specific assessment approaches: (1) iterative pre- and post-tests to measure both content knowledge and affective variables such as attitude and confidence, and (2) diaries to capture insights into cadets' field experiences.

In an extraordinary assessment measure, the external advisors decided to live in the field with the first contingent of cadets in the summer of 1994. In doing so, they gained first-hand knowledge about the nature and scope of the program, and they inadvertently served as assessment instruments themselves. Brig. Gen. Swint remembers well when the consultants came to him with a warning of a "cadet mutiny," which he averted only after a candid two-hour meeting where he explained the rationale for restrictive leave policies that gave the cadets more field experience (wind-blown tents and cold showers!) than they had expected.

Focus groups. Assessment of the FERL program followed another recommended practice: "a good program review process is systematic" (Kleniewski, 1). It is also on-going. Thus, after five years of using the recommended tests and course diaries, the department concluded, according to Maj. Malone (2003) that, "We already knew what we were going to get from these two instruments," and "We were looking for a steady-state assessment tool that was less intensive but just as effective." The tests and course diaries were so labor intensive to process and analyze, he confessed, that the papers sometimes lay untouched for months. The department found their desired longitudinal tool--and much-needed report preparation and analysis--in highly structured focus groups conducted and analyzed by USAFA's Center for Educational Excellence (CEE).

Focus groups for FERL began in 1998 when CEE staff members and trained faculty volunteers from other departments piled out of Air Force vans to set up portable recording equipment in three primitive Quonset huts. Using a protocol that involves an index card activity and a roundtable/ranking activity in addition to open-ended questions (Millis 2003), focus groups continue to this day. In fact, on 6 May 2003, the department discussed "focus group feedback and what we are doing about it" (FERL Briefing 2003). Cadets identified some of the weaknesses such as mandatory dinners (dinners remain required to avoid food waste); cadet leadership (the cadet cadre now sleeps in tents at the FERL site, leading by example); and lack of grading feedback (grades are now posted weekly). Some strengths of the FERL program were the "hands-on activities," "the practical/applicable subjects," and the "heavy equipment."

These strengths were all part of the original dream, a dream affirmed by remarks made by Dr. Ernest L. Boyer, then-president of the Carnegie Foundation for the Advancement of Teaching, at the May 1994 USAF Academy's Outstanding Academy Educator Awards. Linking the FERL program to the scholarship of application, he said, "Theory to practice--practice back to theory. As we make this interlocking relationship, theory in itself becomes more informed through the application of practice as demonstrated in your own Department of Civil Engineering" (Boyer 20).

Assessment successes

Many of the successful assessment components of the FERL program are also evident at the departmental level in USAFA's department of management, a department with thirty-three faculty members offering a popular degree attracting roughly 200 majors a year. Like the FERL program, the management department's assessment approaches involved departmental buy-in and a strategic assessment plan that included self-studies and external consultants.

The leadership qualities of a new management chair prompted departmental buy-in for assessment. Col. Rita Jordan, upon her arrival in 1996, immediately addressed a common fear in virtually all schools that performance/personnel data, such as student course critiques, would be misused. These early years created challenges. At one point, Col. Jordan declared, "The department can have my resignation if I ever learn of intentional mishandling of performance data." Developing trust was much more challenging than developing assessment instruments and schedules.

The assessment instruments and schedules are captured in a systematically administered assessment suite measuring teaching, scholarly contributions, and service: (See Table 1)

A suite of assessment tools ensures a range of data that can be triangulated as needed to support decision making. As an analogy, each instrument provides a snapshot, and the combined suite creates a multidimensional department video. There is a deliberate balance between external and internal instruments and qualitative and quantitative data. For example, the Educational Benchmark Institute Undergraduate Satisfaction Survey (EBIUSS), an external quantitative survey, provides a national benchmark, comparing the department to over 140 schools of management. These data can be compared with qualitative internal focus group data collected from majors close to graduation.

An example of decision making, prior to using internal qualitative focus groups, involved two external instruments and subsequent departmental action. In 1998, despite being satisfied with the overall curriculum, students reported on the EBIUSS a weakness in providing an International Perspective (ranked #108 of 141 schools). The weakness was consistent with a 65th percentile International Business result in the Major Field Achievement Test (MFAT). Our department head directed improvement. We subsequently developed a new course, revised the overall curriculum to reflect globalization, and produced this result: three years later our ability to present International Perspectives was ranked #1 of 183 schools, and the 2001 MFAT results indicated student performance at the 99th percentile.

Another example of decision making was based on internal assessment instruments: the core management course was radically altered after focus group and course critique data pointed to an overlap between leadership topics taught in this course and a course in another department. During a spring 2000 focus group, students noted that the leadership content was repetitive and superficial: It was "skimmed over ... and we do not learn to apply it as we do in our core behavioral science course." A review of the course critique data for the core management course revealed "relevancy" as a steadily declining measure over the previous four semesters. Management instructors met with the behavioral scientists to review course goals and to discuss areas of overlap. Suggested course revisions, discussed and adopted within the department, resulted in positive focus group comments and course critique data in spring 2003, with no mention of repetitiveness or irrelevancy. Thanks to ongoing assessment, the department assessment video returned to clearer focus.

As with the FERL program, external consultants and internal self-studies also provide balance and depth in assessment. Throughout the year, Air Force leaders, industry experts, graduates, and stakeholders visit the department.

Self-assessment occurs, as it does in all departments, through a USAFA-mandated Unit Major Program Review (discussed later), and through the requirements for external accreditation, in this case the Association to Advance Collegiate Schools of Business International (AACSB International). AACSB lauded assessment efforts and in 2002 awarded the USAFA department of management full accreditation the first time the department applied for it (or to use a Western metaphor, the first time "out of the chute"). Self-assessment is appropriately an ongoing norm for this department as well as for others at USAFA.

Overall program assessment efforts

A program evaluation serves an institution by collecting, analyzing, and interpreting information for decisions about whether programs are effective, with the costs and benefits warranting continuation, modification, or elimination. Program evaluation is also a scientific process of collecting reliable and valid evidence (Rossi and Freeman 1993). Decisions should be based on the convergent patterns in the evidence, much as a courtroom jury must sift through complex and often conflicting evidence to reach a verdict.

Our approach to assessment has been guided by our own institutional identity, purposes, and values. We believe that our educational practices must derive from the fundamental values and goals expressed in our institutional vision, mission, core values, development policy, and seven educational outcomes (integrated fundamental knowledge, intellectual curiosity, effective communication, ability to frame and resolve ill-defined problems, ability to work cooperatively with others, the ability to learn independently, and the ability to apply knowledge to the military profession). We believe that assessment is best understood and implemented as a vehicle for both educational improvement and public accountability, and that it must be integrally related to other key activities such as strategic planning, curriculum change, and faculty development. Therefore, we believe our assessment work should be "owned" to the extent possible by the faculty, but it should also be "owned" by senior Academy officials with responsibility for broad educational guidance and institutional leadership. Assessment activities should reflect a diversity of methods, measures, and respondents, and they should address educational practices as well as educational outcomes per se.

This approach is showcased in USAFA's Assessment Catalog (Sheffel, Revak, and Millis 2000, 4-5), now in its second edition, a printed and electronic database providing an in-depth summary of:

   Assessment efforts related to cadet academic
   performance, performance of Academy
   graduates in the Air Force, and performance
   of the departments and agencies in meeting
   the Academy's academic and institutional
   missions ... It allows for flow of assessment
   ideas between and among departments and
   agencies; identifies internal and external
   sources of assessment data; allows for easy
   identification of qualitative and quantitative
   assessment methods; categorizes assessment
   instrument types; tracks the currency
   and frequency of use of assessment methods;
   identifies decisions based on assessment
   data; provides judgments about the utility of
   the assessment methods (low, moderate,
   high); and identifies knowledgeable points
   of contact within departments and agencies.

The Assessment Catalog revealed an impressive variety of assessment measures, many mapped to the educational outcomes, but it also suggested some weaknesses: (1) a failure to share data across departments; (2) an overemphasis on assessment (too much was being done too often--usually to cadets--with negligible impacts on decision making; and (3) an uneven array of assessment instruments including some departments' over reliance on quantitative measures and a USAFA-wide failure to capitalize on nationally normed instruments that would allow comparisons between the Academy and other military institutions or other colleges and universities.

To address these shortcomings, in 2000 USAFA formed the Committee for Excellence through Assessment (CETA) chaired by the director of academic assessment. CETA developed a comprehensive, systematic, ten-year assessment cycle that balanced assessment instruments (internal/external, quantitative/ qualitative, etc.), closed gaps, and set up collection intervals that avoided over-sampling and allowed time for "data collection, analysis, discussion, decision making, and planning" (Revak 2003, 3). The Air Force Academy's Assessment Plan, including the cycle timeline, a description of the instruments, and a "benchmark" history of assessment at USAFA can be accessed at www.usafa.af.mil/dfe/assessment_plan.htm.

At the same link, interested readers can also view the criteria for USAFA's Unit Major Program Reviews (UMPRs). The UMPRs, which are completed every five years, offer a comprehensive review of each department's and staff agency's programs. They provide data for possible mid-cycle "corrections" and for the ten-year self-study for The Higher Learning Commission of the North Central Association of Colleges and Schools (NCA). The UMPRs' required content, which is patterned after the NCA patterns of evidence, discusses program features such as (1) changes to a department's core course(s) and majors programs, including when, how, and why changes were made and efforts to assess the results of those changes; (2) assessments of cadet performance; (3) any changes, such as in equipment/facilities or personnel, that would impact the department's mission; (4) programs for faculty development, enhancement, and evaluation; (5) cadet and faculty research; and (6) significant accomplishments, challenges and goals. The UMPRs, which are reviewed by a committee of senior faculty who report to the dean, are used for institution-wide decision making.

Assessment programs at USAFA, whether institution-wide or at the departmental level (Management) or unit level (FERL), seek to follow "best practices" while remaining true to the intuitional goals, missions, and values. Thus, the measures are varied along many continuums, and the process involves a wide array of individuals from inside and outside the U.S. Air Force Academy. Program assessment is guided, like many other aspects of Academy life and work, by the three core values of "integrity first, service before self, and excellence in all we do."

TABLE 1: DFM ASSESSMENT SUITE

                                       Intellectual
Instrument                  Teaching   Contribution   Service   Faculty

EBI undergraduate
  satisfaction survey           X                        X
Major Field Achievement
  Test--MFAT                    X
Dean of Faculty course
  critiques                     X                                  X
CEE focus groups                X
Management graduate
  survey                        X                        X
Stakeholder interviews          X            X           X         X
Number of management
  majors                        X                        X         X
# Presentations                              X                     X
# Publications                               X                     X
# Consultations &
  participation                              X                     X
% faculty involvement                        X           X
DF Organizational climate
  survey                                     X                     X
Academic promotion rates                                           X
Military promotion &
  feedback                      X                                  X
Faculty development
  program                       X            X                     X
Mid-year feedback               X            X           X

Greater Expectations: The Commitment to Quality as a Nation Goes to College is a multi-year AAC&U initiative to define the aims of a twenty-first century undergraduate education and to discover the best strategies for achieving those aims.

Fifth in a series of articles emerging from AAC&U'S national initiative, Greater Expectations: The Commitment to Quality as a Nation Goes to College. Liberal Education is featuring a series of articles responding to the variety of issues raised in the Greater Expectations initiative about tire future of undergraduate education in America.

Previous articles in the series:

High School-College Connections

Designing Institutional Change

Faculty Development for Teaching Innovation

Liberal Education: Why Now? Why for All?

To respond to this article, e-mail [e-mail], with the author's name on the subject line.

WORKS CITED

Boyer, E. 1996. In Civil engineering U.S. Air Force Academy Quality Air Force Unit self assessment. 1996. USAF, Colorado: Civil Engineering Department.

Civil engineering U.S. Air Force Academy Quality Air Force Unit self assessment. 1996. USAF, Colorado: Civil Engineering Department.

FERL Update: 6 May 03. 2003. Briefing in department of civil and environmental engineering, U.S. Air Force Academy.

Kleniewski, N. May 2003. Program review as a win-win opportunity. AAHE Bulletin. www.aahebulletin.com/member/articles/win-win.asp?pf=1 Downloaded 5/5/2003.

Malone, M. 2003. Private interview with assistant professor and assessment coordinator for the department of civil and environmental engineering, conducted 7 May 2003. USAF, Colorado.

Millis, B. 2003. In Press. A versatile interactive focus group protocol for qualitative assessments. To improve the Academy. Professional and Organizational Development Network in Higher Education. Bolton, MA: Anker Press.

Revak, M. 2003. In Press. Launching a comprehensive assessment plan at the United States Air Force Academy. Assessment Update.

Rossi, P. H. and H.E. Freeman. 1993. Evaluation: A systematic approach. Newbury Park, CA: SAGE Publications, Inc.

Scheffel, D., M. Revak, and B. Millis. July-August 2000. Cataloging the assessment effort at the United States Air Force Academy: Walking the talk for a successful accreditation visit. Assessment Update, 12: 4, 4-5.

Swint, D. 1996. Historical perspective. In Civil Engineering U. S. Air Force Academy Quality Air Force Unit Self Assessment.

Swint, D. 2003. Private Interview with the former head of the department of civil and environmental engineering for eighteen years, conducted 9 May 2003, Colorado Springs, CO.

BARBARA J. MILLIS is director of faculty development, JAMES K. LOWE is associate professor of management/operations research, and ANTHONY J. ARETZ is director of academic strategic planning, programs, and assessment, all at the United States Air Force Academy. This article belongs in the public domain.