Linking Outcomes Assessment with Teaching Effectiveness and Professional Accreditation.

Abstract

One of the most important purposes for the teaching function in a graduate professional school is to foster students' mastery of the programmatic learning outcomes that have been agreed upon by faculty as part of the strategic planning process. Students' learning is a composite of courses they have taken. If the contribution of each course to the achievement of program-level outcomes is an important purpose of teaching, then why is that contribution not routinely assessed as part of annual evaluation of teaching effectiveness? The purpose for this paper is to describe a rationale for viewing course-level learning outcomes as one measure of teaching effectiveness and a component of programmatic outcomes assessment. This paper includes a model for unit-level outcomes assessment that links an individual faculty member's coursework contributions with programmatic outcomes, and describes a strategic planning process for developing faculty consensus on accurate statements of unit-level student learning outcomes and corollary methods of assessment.

Introduction

It is difficult to tune into a political discussion about education in which the term accountability is not used. Although the current debate is primarily centered on public K-12 education, the accountability movement in the past ten years has extended to higher education as well. Regional accrediting agencies and professional societies have refocused their assessments of institutions and professional schools from checklist-style evaluations, toward evaluation of an organization's process for determining desirable outcomes, measuring how well those outcomes are achieved, and using results for improving programs. The general term now applied to this continuous-improvement cycle is outcomes assessment, and for clarity we repeat the idea that it is more than the act of measuring outcomes; it is a system that includes defining programmatic outcomes, measuring programmatic effects, and applying results for guiding and improving programs.

The School of Library and Information Science at the University of South Florida is engaged in a Pilot Study on Alternate Measures of Teaching Effectiveness as a participant in the USF Harvard Grant Project, "Innovations in Faculty Work Life." The School is the only unit in the USF Harvard Challenge Grant Project that is concentrating on the assessment of student learning outcomes and teaching effectiveness for graduate professional curricula. Faculty in the School proposed participation in the Challenge Grant Project for several reasons.

First, faculty were investigating alternatives to the School's process for annual review of teaching and were curious about whether students' learning outcomes could be related to the evaluation process. Second, a revision had been undertaken in the School's comprehensive examination process, and that necessitated a review of programmatic content frameworks and corollary mission, goals, and objectives. These reviews were timely given upcoming professional and regional accreditation visits from the American Library Association (ALA) and the Southern Association of Colleges and Schools (SACS). Both accrediting agencies require a system of outcomes assessment and continuous programmatic improvement, so the Challenge Grant Project was an opportunity to accomplish multiple strategic planning priorities within the School.

The paper provides a brief review of outcomes assessment in higher education and examines the potential for meaningful linkages among student learning outcomes, programmatic outcomes assessment, and teaching effectiveness. The paper includes a model for unit-level outcomes assessment that links an individual faculty member's coursework contributions with programmatic outcomes, and it concludes with a strategic planning process for developing faculty consensus on accurate statements of unit-level student learning outcomes and corollary methods of assessment.

Outcomes Assessment in Higher Education

Neither the philosophy nor the tools of outcomes assessment are new. Business, industry, government, and the military have used strategic planning and quality management tools for years, and education has borrowed and adapted many of those tools. Education, however, has its own history of thinking along outcomes assessment lines. In the mid 1950s Benjamin Bloom and his colleagues published The Taxonomy of Educational Objectives(1) as a framework for classifying student learning outcomes, and behavioral psychologists and the programmed instruction movement gave us performance objectives.

In the 1960s some educational psychologists began to take a systems view of teaching and learning in which a set of components interacted to produce learning outcomes. These components included goals and objectives, the learners, the teacher or learning manager, instructional materials, student performance assessments, and the learning environment. A critical systems concept included by these "instructional designers" was the idea of feedback; that is, system performance could be improved by collecting data and feeding it back into the system to regulate and refine the system.(2) Data typically collected included student performance, student attitude, and information about instructional management and logistics. Measurement and evaluation specialists Cronback and Scriven saw the need to distinguish this type of evaluation for improving learning outcomes and described what has come to be known as formative evaluation. In describing the concept of formative evaluation, Cronback and Scriven contrasted it with summative evaluation, the process of collecting data and information in order to make judgments about the effectiveness of instructional products and programs that are in their complete, final form. In some cases summative evaluation is also used to make decisions about the acquisition or continued use of educational programs or delivery systems.

In 1967 Scriven, Tyler, and Gagne made functional distinctions between summative and formative evaluation that are still used today.(3) The concepts of instructional systems and formative evaluation are largely what we now call outcomes assessment, and pieces of this thinking can be found in the minimum competency testing movement of the 1970s, the standards movement of the 1980s, and the educational accountability movement of the 1990s.

It was not until the latter 1980s that the focus of assessment in higher education evolved toward accountability. For the past 15 years a number of practical projects have been reported that examine the purposes and processes of outcomes assessment. Two prominent leaders in classroom assessment research have been Thomas A. Angelo and K. Patricia Cross who began developing a "Teaching Goals Inventory" in 1986 as a first step in classroom assessment. The first edition of Classroom Assessment Techniques: A Handbook for Faculty(4) was published in 1988. Over the next four years in their Classroom Research Project, they found research on student learning outcomes objectives, but a lack of research or guidelines for helping educators take the next step toward implementing an outcomes assessment process. A second edition (1993) reflected the work of the Classroom Research Project.(5)

Pervasive changes in the higher education environment, such as decreased funding and distance learning, have led to serious consideration of the changing roles of faculty in higher education. Responding to these changes, the Pew Charitable Trusts have funded a number of projects for examining faculty roles and quality assurance practices in higher education, and The American Association for Higher Education (AAHE) began its Forum on Faculty Roles & Rewards in the early 1990s and later branched out with the Assessment Forum. The literature and conferences from both forums include a focus on the role of teaching faculty in the assessment process at both unit and institutional levels, and the role of the assessment process in the evaluation of teaching.

John Wergin is an AAHE senior scholar and has been an active researcher for more than twenty years in the areas of faculty roles, effectiveness, and development. In a study conducted under a grant from the Pew Charitable Trusts, Wergin and his associate Judi Swingen's purpose was

   to search for evaluation policies and practices that encourage both
   constructive change in departments and a stronger culture of collective
   responsibility there. Our goal was to see how these models worked, what
   seemed most critical to their success, and how key ideas might be applied
   to other settings.(6)

The researchers observed that campus practices of quality assurance at the departmental level usually suffer from two debilitating problems:

   Most departments and faculty do not see the relevance of such practices to
   the work they do. The notion of continuous quality improvement hasn't taken
   hold. Further, faculty view institutional measures of quality as off the
   mark, as not congruent with what their own definitions of quality might be.
   Consequently, most program review and outcomes assessment exercises have
   only marginal impact .... There is little coordination at most institutions
   among assessment, program review, and external accreditation.(7)

Wergin and Swingen conclude that at the institutional level demands for unit accountability should focus less on accountability for achieving results and more on how well units conduct evaluations for themselves and use the data these evaluations generate. The study found several examples of "continuous quality improvement" at the departmental level, but few seemed to be able to make the idea work.(8) Faculty members are typically rewarded according to standards of quality dictated by their disciplines, not by standards specific to their institutions or departments.(9) Although linkages between institutional and departmental standards for accountability are the ideal sought in regional accreditation, the mechanisms for creating and maintaining such linkages are seldom formalized and followed.

Wergin and Swingen propose that

   What is needed is a different kind of culture, one in which faculty members
   are able to focus their efforts on activities which best draw upon their
   own skills, talents, interests, and experience, and which allows them to
   negotiate with their colleagues how they might best use these strengths to
   contribute to the work of the department. Thus, some faculty might put
   relatively more effort into research, others into teaching, still others
   into institutional and professional service; and these areas of emphasis
   could shift throughout the course of a career. But this kind of
   restructuring will be possible only when institutions shift the unit of
   analysis from the individual to the academic department, when the work of
   the department is evaluated as a whole, rather than simply as the aggregate
   of individual faculty accomplishments, and when faculty are evaluated
   according to their differential contributions to the group.(10)

In her paper on department-level improvement, Hatfield says that

   the goal of a continuous improvement initiative is for a department to
   become self-regarding, self-monitoring and self-correcting.... Finding out
   how students have fared years after graduation isn't good enough.
   Continuous improvement means that the correction cycle is shorter because
   progress is constantly being monitored. Deficiencies are identified and are
   corrected while the student is still on site, and corrections are made to
   department level processes to keep similar deficiencies from occurring....
   The result is that the department is confident of the quality of their
   graduates' knowledge and skills before the students enter the work
   force.(11)

Department level assessments must be self-renewing. Assessment data and information must feed back into the system, both on the university and departmental level. A department level assessment plan should identify the mission of the department; goals related to that mission; activities or processes supporting the achievement of the goal; and a number of measures which, when taken together, provide indication of the degree to which the goal is being achieved. Implementing the plan requires the collecting, analyzing, and benchmarking of data; revision of processes; and communication of results.(12) These assessment processes have been incorporated into a number of the regional accrediting associations' reporting procedures.

Outcomes Assessment at the University of South Florida

In 1997, in response to ongoing faculty concerns about the adequacy of student evaluations to measure all dimensions of teaching effectiveness, the USF Provost appointed a cross-disciplinary task force to consider the evaluation of teaching. The Committee on the Importance of Teaching did not believe it was possible, nor desirable, to articulate uniform definitions or practices of effective teaching that would be applicable across all disciplines. They did, however, identify four principles that ought to guide the assessment of quality teaching.

* Teaching is a multifaceted process that cannot be measured by a single instrument. Effective teaching practice varies between fields and individuals. As a result, a wider range of data should be considered in the assessment of teaching than student evaluations of teaching.

* Student learning outcomes should be a primary criterion of teaching effectiveness. Departments and faculty should identify their learning goals and demonstrate that those objectives are being met in teaching practice.

* Student learning is more than the sum of its parts. Students' learning experiences transcend individual courses and reflect the cumulative experience of their major and general education courses. As a result, the assessment of teaching effectiveness should include departmental and programmatic units, as well as individual courses. Measures of teaching effectiveness should include the extent to which faculty and departments contribute to the larger unit's goals, as well as fulfilling the objectives of individual courses.

* The results of teaching assessment have multiple purposes: enhancement of individual faculty teaching, improvement of program development and delivery, and the provision of a broader base of material on which to base tenure and promotion decisions.

All four of these principles reported by the Committee on the Importance of Teaching speak directly to the concept of outcomes assessment, the use of assessment data for improving teaching, and the relationship between student learning outcomes and evaluation of the quality of teaching.

In 1998, when USF was selected to participate in the Harvard Challenge Grant(13) Project, it was decided that the Pilot Study on Alternate Measures for Effective Teaching would become one of the foci of the project, and it would build upon the efforts and recommendations of the Committee on the Importance of Teaching.(14) From these meetings four guidelines were formulated for the pilot study.

* The pilot departments/units will incorporate a broad range of data in their faculty evaluation process that reflect the multifaceted nature of effective teaching.

* The pilot departments will focus on student learning outcomes as a primary, but not necessarily exclusive, criterion of teaching effectiveness. The faculty members will first identify the primary goals for both their units and for individual courses. Teaching effectiveness will be measured/demonstrated, at least in part, in relation to these identified student outcomes.

* Because student learning occurs across courses throughout a student's major, pilot departments will articulate unit-wide learning outcomes and measures of teaching effectiveness that reflect the unit's unique qualities. The pilot departments will create frameworks for evaluating individual faculty members on their contributions to the unit-wide goals, as well as their individual course goals.

* The pilot assessment measures should produce data that can be used effectively to enhance individual faculty teaching practices and serve as a means to review and revise the delivery of a department-wide curriculum.

In addition to the initiatives undertaken with the Committee on the Importance of Teaching and the Harvard Challenge Grant, the Provost's Office has also taken a look ahead to the SACS self study that USF will carry out in 2003 and has offered outcomes assessment seminars, developed a standardized outcomes assessment reporting format, and mandated development during the 2000-2001 school year of "mini" outcomes assessment procedures at the department-level throughout the university. Colleges have been charged with managing follow up and reporting.

The School of Library and Information Science Pilot Study

The School followed a carefully-managed strategic planning process for designing its outcomes assessment plan. Three senior faculty members formed a team to manage the pilot study project. The project team believed that broad faculty participation and consensus were required, and also felt that the faculty would need structure and guidance for completing the project tasks. The method used to structure the work was to approach the tasks in developmental cycles wherein the project team created a small sample of the work; submitted it to the entire faculty for review, revision, and consensus; and then divided the main body of work among faculty teams for completion. Completed work was then brought again to the whole faculty for consensus.

* The first step was to develop a model of the outcomes assessment process. The three-member pilot project team created the model and then submitted it to the full faculty for review, revision, and consensus. The team began by identifying characteristics of an outcomes assessment model that would meet the needs of: (1) Harvard Challenge Grant participation, (2) ALA and SACS accreditation, (3) the School's comprehensive exam process, and (4) the School's annual review of faculty.

* To be successful the model would need to: (1) show relationships between the School's mission and goals and USF's mission and goals, (2) be based on a comprehensive programmatic outcomes framework, (3) be data driven, (4) support multiple measures of outcomes, (5) link student learning with teaching effectiveness, and (6) gain consensus faculty support. It was also hoped that the model would be exportable for use by other departments and schools. The model shown in Appendix A (available at <http://luna.cas.usf.edu/~carey/aeq>) was developed to meet these criteria. It begins with USF's mission and goals and links them with the School's mission and goals. Note that three SLIS goals are indicated, but only the one for teaching is carried through the entire model. The model then depicts the relationships among student learning outcomes, outcomes assessment, evaluation of teaching effectiveness, and use of assessment results for program improvement. This model became the conceptual base for the work that followed.

* In the next step the three-member pilot project team revised the School's mission and wrote a goal for teaching. Then the Program Content Framework in Appendix B (available at <http://luna.cas.usf.edu/~carey/aeq>) was designed to guide development of student learning outcomes for the teaching goal. This framework outlines six content areas with several topics for each. These were not meant to have direct correlation with existing courses, but were regarded as outlining the knowledge and skills every student should master in the SLIS program. On the right hand side of the Program Content Framework, the vertical threads of "Accepted Practices," "Trends and Issues," and "Networks, Systems, and Technology," are to be viewed as infusing all six content areas. To finish this step, the revised mission, the goal for teaching, and the content framework were all submitted to the full faculty for review, revision, and consensus.

* The three-member pilot project team then chose one of the teaching goal areas and wrote sample student learning outcomes. The sample outcomes were written to gain consensus from faculty on the development process and to provide a model for broader faculty participation in developing student learning outcomes for all six teaching goal areas. The work at this stage was reported on the form in Appendix C (available at <http://luna.cas.usf.edu/~carey/aeq>). The form establishes the linkage between USF's mission and goals and the School's goals, and lists specific student learning outcomes within each goal.

* Six faculty work teams were formed to write student learning outcomes for the six teaching goal areas. After completing their work, the six teams of two-three faculty members brought their student learning outcomes to the full faculty for review, revision, and consensus. In a meeting of the full faculty, agreement was achieved on the conceptual content of the learning outcomes for all six of the content areas, but details of final sequencing and wording were assigned to a subgroup for completion. The subgroup's work was submitted at a later time and approved by the full group.

* While step 4 was in process, a subcommittee of four faculty members prepared an inventory of assessment methods that was subsequently edited by the full faculty. The inventory was then incorporated into the matrix illustrated in Appendix D (available at <http://luna.cas.usf.edu/~carey/aeq>). Note that the first five "Means of Collecting Data" are programmatic measures that could be applied across all courses, while the rest of the means of collecting data are measures that would be applied within a course.

* The matrix in Appendix D covers "Professionalism" and "Information Services," two of the six teaching goal areas. It and similar matrices for the other four goal areas were distributed to faculty members who had appropriate content expertise and teaching experience. Faculty were encouraged to select multiple means of collecting data for each objective as appropriate for the outcomes. The results of this stage were consolidated onto the form in Appendix E (available at <http://luna.cas.usf.edu/~carey/aeq>) which provides for recording the assessment measures used; briefly describing assessment results; and most importantly, in the last blank, the "Use of the Results to Improve Instructional Program." It is in the use of the results that the student learning outcomes assessment data are applied to improving teaching and programmatic effectiveness. These reporting forms will be completed as assessments are implemented and results are utilized.

Discussion

At this point in the project, the School has developed and implemented a model for faculty consensus on programmatic outcomes in the goal area of teaching. Although much has been completed and the Harvard Pilot Study is drawing to a close, the School has continued the strategic planning process and is using its consensus model to define programmatic outcomes and assessment measures for the School's goal areas of research and service. After outcomes are written and assessments are specified, the School will develop methods and instruments for exit surveys, employer surveys, and alumni surveys that cover teaching, research, and service outcomes. Then, of course, assessments of outcomes must be carried out systematically and the results must be utilized for programmatic improvement. Finally, faculty concur on the principle that student learning outcomes will be linked to annual review of faculty members' teaching effectiveness, but this principle must be implemented through new governance policies and guidelines on which faculty can agree.

The full faculty actively participated in the pilot study process over a period of six months. The three-member project team was pleased with the pace, results, and faculty buy-in to the purposes for the project. In retrospect, it was worth taking time to develop the conceptual model in Appendix A to channel group thinking, and providing work samples for the faculty teams before they began their productive efforts helped to steer the work and streamline the process. It was also helpful to keep meetings of the full faculty focused on conceptual decisions and assign the more editorial functions of wording and sequencing to smaller subgroups of the faculty.

The strategic planning efforts have already had payoff for the School, providing material for a College of Arts and Sciences planning assignment and for the School's Biennial Narrative Report to the Committee on Accreditation of the American Library Association. The School has also used the Program Content Framework in Appendix B for new student orientation, adjunct faculty briefing, and revision of the master's degree comprehensive examination. Once the outcomes assessment process is in place, it will become central to the School's reporting for regional and professional accreditation.

Conclusions

The outcomes assessment approach emphasizes student learning outcomes at both the course and programmatic levels, and measures those outcomes course-by-course, at graduation, and during students' professional lives. This shifts the view of the School's teaching mission from a narrow focus on each faculty member in assigned classes to the broader perspective of how well the School is preparing students for careers and how different faculty members are contributing to that preparation. In the outcomes assessment approach, the questions of accountability are clarified. The School is accountable to the institution and the profession for the effectiveness of the program, and to the graduates for their preparation for careers in library and information professions. Accountability for outcomes on a course level becomes an "internal" concern; that is, the School and individual faculty members take joint responsibility for the quality of courses and their contributions to the program. Under an outcomes assessment approach, both the School and its faculty must define the exercise of their professional responsibilities in teaching, research, and service as being forever formative; that is, information must be gathered continuously to support decisions about improving what we do for our students, our institution, and our profession. The responsibility for ensuring implementation of an outcomes assessment approach, however, does not rest solely with departments and their faculty; rather, the institution must become a partner in the outcomes assessment process by providing the policies, the resources, and the reward structure that encourages and supports department and faculty participation.

Endnotes

(1) Benjamin Bloom, et al. Taxonomy of Educational Objectives: The Classification of Educational Goals: Handbook 1; The Cognitive Domain. (New York: W. H. Freeman, 1956).

(2) Walter Dick and Lou Carey. The Systematic Design of Instruction. (Glenview, IL: Scott Foresman, 1978).

(3) Michael Scriven, Ralph Tyler, and Robert M. Gagne. Perspectives of Curriculum Evaluation. AERA Monograph Series on Curriculum Evaluation. (Chicago: Rand McNally, 1967).

(4) Thomas A. Angelo and K. Patricia Cross. Classroom Assessment Techniques: a Handbook for Faculty. (Ann Arbor: National Center for Research to Improve Postsecondary Teaching and Learning, University of Michigan, 1988).

(5) Thomas A. Angelo and K. Patricia Cross. Classroom Assessment Techniques: A Handbook for College Teachers. 2nd ed. (San Francisco: Jossey-Bass Publishers, 1993).

(6) Jon Wergin and Judi N. Swingen. Evaluating Academic Departments: Best Practices, Institutional Implications. In press. (Washington, D.C.: American Association for Higher Education, 2000).

(7) Ibid

(8) Ibid

(9) Jon Wergin. "Assessment of programs and units," in Architecture for Change, Proceedings of the AAHE Assessment Conference, June 13-17, 1998, Cincinnati, OH: 59-65, p. 62.

(10) Ibid

(11) Susan R. Hatfield. "Department Level Assessment: Promoting Continuous Improvement." IDEA paper #35, May 1999 (IDEA Center, Kansas State University): [1].

(12) Ibid

(13) USF began participation in 1998 in "Innovations in Faculty Work Life," a Pew Trust Challenge Grant project administered by the Harvard University School of Education. USF is one of seven institutions in the project along with Olivet College, University of North Carolina, Arizona State University, University of South Dakota, Tulane University, and Seattle Pacific University. The initial meetings for the teaching effectiveness group took place in March/April 1999.

(14) University of South Florida. "The Importance of Teaching Project." 1998. <http://acad.usf.edu/cvisot/teachindex.html>. [Editor's Note: This link appears to be outdated.]

James O. Carey, University of South Florida
Anna H. Perrault, University of South Florida
Vicki L. Gregory, University of South Florida

Dr. Carey, Assistant Professor, School of Library and Information Science, teaches courses in instructional technology and school media management <[e-mail]>. Dr. Perrault, Associate Professor, School of Library and Information Science, teaches courses in research methods, government and humanities information sources, collection assessment, and academic libraries <[e-mail]>. Dr. Gregory, Professor, School of Library and Information Science, teaches courses in collection development, information science, and library networks and systems: <[e-mail]>.