Assessing the Effectiveness of Problem-Based Learning in Higher Education: Lessons from the Literature.

Abstract

Problem-Based Learning (PBL) is an innovative educational approach that is gaining prominence in higher education. A review of the literature of PBL outcomes summarizes, across multiple studies, the positive effects of problem-based learning. Since PBL brings with it unique challenges to traditional assessment, however, this study suggests alternative approaches. Alternative assessment may provide additional insight into the effectiveness of PBL and other alternative pedagogies.

Introduction

Traditional instruction, such as the typical lecture-based session that developed before textbooks were mass-produced, often involves delivering as much information as possible as quickly as possible. The lecture method was one of the most effective and efficient ways to disseminate information and has often been used for this end. Because many faculty members are poor lecturers, and because students are often poor participants in the lecture, this type of instruction has often allowed students to be passive in the classroom. Students, not knowing how to be active participants in the lecture, have relied on transcription, memorization, and repetition for learning.

In recent decades, however, we have learned a great deal from cognitive science research about the nature of learning. Students construct knowledge; they do not take it in as it is disseminated, but rather they build on knowledge they have gained previously (Cross, 1998). They benefit from working together, and they may learn best from teaching each other (Annis, 1983; McKeachie, et al., 1986). Research also suggests that students learn best in the context of a compelling problem (Ewell, 1997); they learn through experience (Cross, 1999). In short, students learn through making cognitive connections, social connections, and experiential connections (Cross, 1990). Because they make these connections differently, students do not learn in the same way. This relatively new information suggests that teaching is a complex activity, and it necessitates the emergence and development of approaches to instruction that are consistent with what we know about the way that learning happens (Ewell, 1997). This new understanding has given rise to the notion of a paradigm shift in higher education, one from a focus on teaching to a focus on learning (Barr and Tagg 1995). New "powerful pedagogies" emphasizing learning, such as project-based learning, inquiry-based learning, case-based learning, research-based learning, situation-based learning, action learning, and Problem-Based Learning (PBL) intimate that alternative pedagogies may be gaining in prominence and may ultimately become the dominant classroom paradigm.

PBL is an educational approach in which complex problems serve as the context and the stimulus for learning. In PBL classes, students work in teams to solve one or more complex and compelling "real world" problems. They develop skills in collecting, evaluating, and synthesizing resources as they first define and then propose a solution to a multi-faceted problem. In most PBL classes, students also summarize and present their solutions in a culminating experience. The instructor in a PBL class facilitates the learning process by monitoring the progress of the learners and asking questions to move students forward in the problem-solving process. Unlike traditional classrooms, the faculty member is not the sole resource for content or process information, but instead guides students as they search out appropriate resources.

The PBL approach had its start in the 1960s at McMaster Medical School as faculty developed PBL out of the perceived need to produce graduates who were prepared to deal with the information explosion, and who could think critically and solve complex problems. This institution developed its entire curriculum around problem-based learning. Soon after medical schools around the world began to adopt the McMaster model. In these cases, PBL is an approach to structuring the curriculum that involves confronting students with problems from practice which provide a stimulus for learning (Boud & Feletti, 1991). However, there are many possible forms that a curriculum and process for teaching and learning might take and still be compatible with this definition (Boud & Feletti, 1991). For example, educational and professional schools also began to feel many of the same needs as medical schools, so they began to adopt the approach as well, although in different forms, such as hybrid PBL and traditional curricula and course-by-course models; again the approach spread to institutions around the world.

Finally, educators and employers alike began to call for change in undergraduate institutions (Jones, 1997). They also wanted students who could think critically, solve problems, and work in teams. The 1998 Boyer Report, Reinventing undergraduate education: A blueprint for America's research universities, for example, articulates these changes and recommends inquiry-based learning as a vehicle for improvement (The Boyer Commission, 1998). And many undergraduate institutions began to develop PBL programs and curricula. Aalborg has one of the most comprehensive undergraduate PBL curriculums, and Maastricht also has a developed PBL program of study. More recently, in the U.S., the University of Delaware has turned attention toward PBL, as has Samford University in Birmingham, Alabama. In addition to these more comprehensive efforts, individual faculty members at more than 300 institutions are using PBL at the undergraduate level (PBL Insight, p. 7).

In developing expectations about the outcomes that PBL can bring, we can learn from the research that has been conducted in medical schools. Therefore, in this article, we review recent literature on PBL from medical and professional schools that outlines PBL effectiveness compared to traditional instruction. We also review recent literature from professional schools and undergraduate programs, which suggests that traditional assessment may not be adequate to evaluate the effectiveness of PBL as an instructional method. This research shows that PBL provides students with the opportunity to gain theory and content knowledge and comprehension. In addition, PBL helps students develop advanced cognitive abilities such as critical thinking, problem solving, and communication skills. PBL also can improve students' attitudes toward learning. Thus, as a pedagogical technique, problem-based learning promotes the kinds of active learning that many educators advocate (Barr & Tagg, 1995).

Literature

Several studies focus on the change in knowledge and skill levels that occur with PBL instruction. A few studies show slight decreases in knowledge of basic sciences (Albanese & Mitchell, 1993). Other studies show that on tests of medical knowledge, students in traditional programs scored higher than students in the PBL curriculum (Schmidt, Dauphinee, & Patel, 1987; de Vries, Schmidt, & de Graaff, 1989). Overall, most studies show no significant difference between the knowledge that PBL students and non-PBL students acquire about sciences (Albanese & Mitchell, 1993). However, students who acquired knowledge in the context of solving problems have been shown to be more likely to use it spontaneously to solve new problems than individuals who acquire the same information under more traditional methods of learning facts and concepts through lectures (Bransford, Franks, Vye, & Sherwood, 1989). In addition, students in the problem-based learning environment have developed stronger clinical competencies although the differences were small and non-significant (de Vries, Schmidt, & de Graaff, 1989). A study conducted in a nutrition and dietetics course found that PBL students perceived that they developed stronger thinking and problem-solving skills, effective communication skills, and sense of personal responsibility than did students who received lectures (Lieux, 1996).

Much of the medical school research shows that student attitudes toward learning do change. Students in PBL courses often report greater satisfaction with their experiences than non-PBL students. For example, PBL medical students at Harvard reported their studies to be more engaging, difficult, and useful than did non-PBL students (Albanese & Mitchell, 1993). Additional research studies document that students who experience PBL have substantially more positive attitudes toward the instructional environment than do students in more traditional programs. PBL students tend to give high ratings for their training whereas students in traditional programs are more likely to describe their training as boring and irrelevant (de Vries, Schmidt, & de Graaff, 1989; Schmidt, Dauphinee, & Patel, 1987). These changes in attitudes are marked by an impact on student retention. In countries with high dropout rates among medical school students, such as in Holland, students in the PBL medical program were much more likely to graduate and do so in less time than students in the more traditional curriculum (de Vries, Schmidt, & de Graaff, 1989). In addition, attendance was significantly higher in the PBL class than in the lecture version (Lieux, 1996).

Research also shows changes in student study behavior. Coles (1985) and Newble and Clark (1986) report that students were more likely to use versatile and meaningful approaches to studying than non-PBL students, who were likely to use reproduction. Nolte, et al., (1988) found that use of reserve material went up. Blumberg and Michael (1992) found that PBL students were more likely to use textbooks and other books and informal discussion with peers than did non-PBL students, who were more likely to rely on lecture notes.

Research of PBL in medical schools, as seen in the reviews by Albanese and Mitchell (1993), Vernon and Blake (1993), and others mentioned above has focused primarily on comparing the outcomes of PBL methods to more traditional pedagogical methods. Research on PBL as a method for preparing professionals has followed in this tradition. These studies do provide insight as to how PBL compares to traditional methods.

However, PBL presents some unique challenges for assessment. Because the focus of this pedagogy is primarily on learning to learn and less on mastery of a particular body of knowledge, traditional methods of course assessment such as examinations may not be very effective (Major, 1999). If traditional assessment is a good measure of traditional pedagogy, it stands to reason that an alternative assessment may be necessarily a better measure for an alternative pedagogy, such as PBL. Using alternative assessment in the case of PBL can help bridge the gap between instruction and assessment. Authentic assessment uses tasks developed from realistic activities in the professional world (Nightingale, Te Wiata, Toohey, Ryan, Hughes & Magin 1996). Nightingale, Te Wiata, et al., (1996) define authentic assessment tasks as "complex simulations, case studies, ... or multi-faceted projects ... assessing a range of knowledge, skills and attitudes in the one assessment task."

Some signs of a movement in this direction exist. Recently studies have begun to investigate PBL outcomes, such as teamwork or presentation skills, that may not be associated with traditional lecture methods. Cockrell, Caplow, & Donaldson (2000), for example, recently conducted a study examining students' perspectives on their leaming as members of collaborative groups. The researchers, using interpretive methods, found that the collaborative groups fostered students' sense of ownership of the knowledge that was created over the semester. The researchers also suggested that within the groups, leadership moved from student to student as situations arose and resolved. More studies like this one are needed to determine the effectiveness of PBL in higher education.

Discussion

These recent studies show that it is time to think outside of the box on how we assess the effectiveness of PBL and how we think about its outcomes. Alternative assessment which is authentic to the leaming environment can have a positive influence. An alternative assessment measure might include constructed response items, essays, writing samples, oral presentations, exhibitions,

experiments, and/or portfolios (Ewing, 1998). In a PBL classroom, these measures might be much more relevant and authentic to a problem-solving setting than a traditional standardized multiple-choice test. Allowing students to engage in these kinds of measures can allow us to assess important learning by examining and judging the students actual or simulated performance on significant tasks (Worthen, 1993).

As with any assessment planning process, the principles of good assessment can help guide practitioners. Instructors should begin by knowing what they want their students to achieve and how they want students to get there. They should consider that learning is a multidimensional activity; including knowledge and abilities as well as values, attitudes, and habits of the mind. In addition, when formulating learning goals, faculty members should think of learning in the larger context of the educational community. Assessment of whether goals have been attained and learning has occurred should have a clearly stated purpose, one related to the learning. Thus, it should focus on comparing educational goals and expectations with performance. In addition, assessment should be ongoing, throughout a semester, rather than occurring only at the end (American Association for Higher Education, 1992).

While each particular PBL instructional environment is unique, and therefore merits its own unique assessment strategy, several alternative assessment techniques seem particularly appropriate for the PBL learning environment.

   Outside Evaluation by Experts. Since many PBL experiences involve a
   culminating experience such as a presentation, written projects, or
   portfolio, systematic evaluation of these capstone projects by a team of
   outside experts can afford one means of assessing student performance.

   Content Analysis of Projects. Because PBL uses ill-structured problems with
   many possible solutions, each student group within a class may pursue a
   slightly different domain of knowledge. To assess the range of content
   knowledge learned by students in the class, instructors may need to
   evaluate across assignments and groups to look for the variety of resources
   students are collecting. Project analyses may also be useful in assessing
   skills such as researching, critical analysis, or writing.

   Focus Groups. One method that can prove useful for assessing outcomes such
   as teamwork or leadership is the use of focus groups. Students can offer
   perspectives on their experience within the problem-solving group and may
   be able to reflect on their own growth across the experience.

   Peer Evaluations. Many PBL instructors use some form of peer evaluation as
   a means of moderating individual student behavior within the group setting.
   However, these evaluations can also be used to gather data about the level
   of skill development across individuals in the class.

   Journals or Activity Logs. Students in PBL classes often do the bulk of the
   work for a project outside the formal classroom. The work completed outside
   the classroom can be difficult to assess, so many instructors require
   students to keep a log or journal of the work they complete for the
   project. Mid-semester and/or end of semester evaluations of these journals
   or logs can provide excellent evidence of student learning throughout the
   project.

   Personal Reflections. For many students, the type of active learning that
   PBL requires is an unusual experience. One method for assisting students in
   their metacognitive understanding of the PBL process is to ask them to
   reflect on the experience of PBL at key points in the process. Qualitative
   analyses of these reflections can offer supportive evidence for many
   process-type outcomes such as developing critical thinking or research
   skills.

These techniques focus on the contextual nature of PBL, requiring the students to produce an authentic product that is related to the problem and to make judgments about their performances.

Conclusion

The academy and its constituents have called for change in the way that teaching and learning are carried out in undergraduate education, and change is happening on many campuses and in many classrooms. Assessment must keep pace with these trends. We must begin to look beyond traditional measures toward new and different ways of knowing. PBL is just one of the many pedagogies that illustrate this fact, but educators' experience with PBL can shed new light on what, why, and how we are doing in the academy.

References

Albanese, M.A. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic medicine: Journal of the Association of American Medical Colleges, 68 (1), 52-81.

American Association for Higher Education (1992). Nine principles of good practices for assessing student learning. Washington, D.C.: American Association for Higher Education.

Annis, L. F. (1983). The process and effects of peer tutoring. Human Learning, 2, 39-47.

Barr, R., & Tagg, J. (1995). From teaching to learning: A new paradigm for undergraduate education. Change, 27 (6), 12-25.

Barrows, H. S. (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for Teaching and Learning, 68, 3-12.

Blumberg, P. & Michael, J. A. (1992). Development of self-directed learning behaviors in a partially teacher-directed problem-based learning curriculum. Teaching and Learning in Medicine, 4 (1), 3-8.

Boud, D. & Feletti, G. I. (1991). The challenge of problem-based learning. London, Kogan.

The Boyer Commission on Educating Undergraduates in the Research University. (1998). Reinventing undergraduate education: A blueprint for America's research universities. New York: The Carnegie Foundation for the Advancement of Teaching.

Bransford, J. D., Franks, J. J., Vye, N. J., & Sherwood, R. D. (1989). New Approaches to Instruction: Because Wisdom Can't Be Told. In S. Vosiadou & A. Ortony (Eds.), Similarity and Analogical Reasoning (pp. 470-297). New York: Cambridge University Press.

Cockrell, K. S., Caplow, J. A. H., & Donaldson, J. F. (2000). A Context for Learning: Collaborative groups in the problem-based learning environment. Review of Higher Education 23 (3), 347-363.

Coles, C. R. (1985). Differences between conventional and problem-based curricula in their students' approaches to studying. Medical Education, 19 (4), 308-309.

Cross, K. P. (1998). Opening windows on learning: The cross papers number 2. Mission Viejo, CA: League for Innovation in the Community College and Educational Testing Service.

Cross, K. P. (1999). Learning is about making connections: The cross papers number 3. Mission Viejo, CA: League for Innovation in the Community College and Educational Testing Service.

de Vries, M., Schmidt, H. G. & de Graaf, E. (1989). Dutch comparisons: Cognitive and motivational effects of problem-based learning on medical students. In Schmidt, H. G., Lipkin, M., de Vries, M. W. & Greep, J. M. (eds), New Directions for Medical Education: Problem-based learning and community oriented medical education. (pp. 230-240). New York: Springer-Verlag.

Ewell, P. T. (1997). Organizing for learning: A new imperative. AAHE Bulletin, 50 (4), 3-6.

Ewing, S. C. (1998). "Alternative assessment: Popularity, pitfalls, and potential." Assessment Update: Progress, Trends, and Practices in Higher Education 10, (1) 1, 2, 11, 12.

Jones, E. A. (1997). Identifying college graduates' essential skills in reading and problem solving: Perspectives of faculty, employers, and policy makers. University Park, PA: National Center on Postsecondary Teaching, Learning, and Assessment.

Lieux, E. M. (1996). A comparative study of learning in lecture versus problem-based format. About Teaching, 50, 25-27.

Major, C. (1999). Connecting what we know and what we do through problem-based learning. AAHE Bulletin, 51 (1), 7-9.

McKeachie, W. J., Pintrich, P. R. Lin, Y.-G, & Smith, D. A. F. (1986). Teaching and Learning in the College Classroom: A Review of the Research Literature. Ann Arbor: University of Michigan, National Center for Research to Improve Postsecondary Teaching and Learning.

National Education Goals Panel. (1992). The national education goals report: Building a nation of learners. Washington, DC: U. S. Government Printing Office.

Newble, D. I., & Clarke, R. M. (1986). The approaches to learning of students in a traditional and in an innovative problem-based medical school. Medical Education, 20 (4), 267-273.

Nightingale, P., Te Wiata, I., Toohey, S., Ryan, G., Hughes, C., & Magin, D. (eds). (1996). Assessing learning in universities. Sydney: UNSW Press.

Nolte, J., Eller, P. & Ringel, S. P. (1988). Shifting toward problem-based learning in a medical school neurobiology course. Proceedings of the Annual Conference on Research in Medical Education, 27, 66-71.

Schmidt, H. G., Dauphnee, W. D., & Patel, V. L. (1987). Comparing the effects of problem-based and conventional curricula in an international sample. Journal of Medical Education, 62 (4), 305-315.

Vernon, D. A., & Blake, R. L. (1993). Does problem-based learning work? A meta-analysis of evaluative research. Academic Medicine, 68 (7), 550-563.

Worthen, B. (1993). Critical issues that will determine the future of alternative assessment. Phi Delta Kappan, 74, (6), 444-448, 450-454.

Claire H. Major, University of Alabama Betsy Palmer, University of Iowa

Major is an assistant professor of higher education administration. <[e-mail]>. Palmer is an assistant professor in the Counseling, Rehabilitation and Student Development department <[e-mail]>.