AUTHENTIC ASSESSMENT OF PROBLEM-BASED LEARNING.

ABSTRACT

In the quest to nurture the whole student, nursing faculty have sought more engaging strategies of teaching and learning. One of these strategies is problem-based learning (PBL), which was developed to improve medical education by moving from a subject and lecture-based curriculum to an interdisciplinary one guided by real-life problems. The primary learning outcomes that PBL achieved were to improve critical thinking, self-directed learning, communication, and interdisciplinary collaboration, enhance ability to contribute to a team, and develop life long learning skills. Quantitative assessments from pre and post essay exams, clinical evaluations and the weekly and semester long PBL problems determined comprehensiveness of the student's knowledge of the concepts in the course and the student's ability to communicate an understanding of the course. Qualitative assessments from the individual papers, clinical journals, peer and course evaluations revealed that these outcomes were achieved.

INTRODUCTION

PBL has evolved from the theory that learning is a process in which the learner actively constructs information (Gijselaers, 1996). Information presented only from a lecture format is of limited use and unlikely to assist students in acquiring the desired outcomes of a baccalaureate nursing education. Yet, until recently, the predominant education theory was one in which learning meant filling students' heads with information (Bruer, 1993). This bucket theory assumed that students stored the given information and regurgitated it upon request.

Currently, students are known to understand and retain information based upon the relationships to pre-existing structures and information (Bruer, 1993; Bruning, Schraw & Ronning, 1995). Nursing education should adapt to this shift in understanding about how learning occurs. The challenge for nursing education has always been to develop initiatives and innovations that will prepare graduates for the rapidly changing healthcare environment. Managing patients with increasingly acute conditions requires nurses with excellent analytical, decision-making, and communication skills. Students often experience difficulty in transferring knowledge from the physical and psychosocial sciences to the clinical practice settings. Current educational objectives may be contributing to the gap between practice and theory. Objectives, which only require memorization, recall, or basic understanding, are not preparing the students to apply knowledge in the clinical arena.

Education is too often teacher-centered, keeping the student in a passive role. Research suggests that the ability to retain knowledge does not inevitably produce a competent practitioner (Barrows & Tamblyn, 1980). Confucius is quoted as saying, "I hear and I forget; I see and I remember; I do and I understand." In PBL students act as professionals and challenge problems as they would occur in a real clinical situation with insufficient information, and a need to determine possible solutions by a given deadline. (Barrows, 1985). Nursing educators know that students learn better when actively involved in the process. Most nursing courses have a clinical (hands on) experience built into the course requirements. Yet, educators have continued to focus on imparting only knowledge in the classroom setting.

For over ten years the nursing literature has underscored the need for a curriculum revolution in nursing education, emphasizing innovative strategies which allow for the socialization of a student as a colleague and professional (Butterfield, 1990; Frost, 1996; Heliker, 1994). Nursing students become socialized in to the skill needed in professional practice through valuing learning, acquiring and accepting information from other sources, critically questioning others, and obtaining feedback on learning outcomes through self, faculty and peer evaluations. (Heliker, 1994). It is in the process of encountering and evaluating real life problems that students learn both content and critical thinking.

The educational philosophy of problem-based learning (PBL) is centered on the belief that students should be actively involved in the process of learning (Barrow & Tamblyn, 1980). PBL is an exciting alternative to the more traditional lecture approach. PBL allows the students to use real-life problems to identify what they know, what they do not know, what other information is needed, where to find that information, how to analyze it, and how to communicate that information to others. The PBL approach is a systematic way of helping students to comprehend the problems of complex situations and to use experience, resources, and theoretical knowledge in solving problems, communicating and collaborating within interdisciplinary fields. By focusing on real-life problems, feedback, class discussion and student reporting, the classroom is transformed into a vibrant, active learning environment. As health care increases its dependence on technology and genetic research in order to serve a more varied cultural, racia l, religious, and aged population, nursing must demonstrate competence in several key areas: critical thinking, communication, assessment, and technical skills. PBL integrated real-life practice problems into the academic aspects of instruction which brought new life to the classroom.

METHODS

In redesigning this course to the PBL format, the research question to be answered was; does the PBL methodology improve or achieve the stated learning outcomes? The learning outcomes to improve critical thinking, self-directed learning, communication, interdisciplinary collaboration skills, enhance ability to contribute to a team and develop lifelong leaning ability. PBL was first introduced in the nursing curriculum in Community Health Nursing, which is a required nursing course in the baccalaureate and past RN mobility programs in the School of Nursing. It is offered the final semester of the senior year and is a co-requisite with the Management and Senior Seminar courses. There were 24 students in the class, 23 females and one male and all were licensed RN's. The age range was 19-49 years. All of the students were full-time students, Caucasian and English speaking. The majority of the students had between a 3.0- 4.0 grade point average and expected to make an A or B in the course. A pre and post PBL asse ssment was done to evaluate learning styles of these students prior to experiencing PBL.

The course met one day a week for a three hours of class with each student scheduling 9 hours a week of clinical practice in a designated community setting. Clinical opportunities occurred in a variety of settings with all students sharing the common experiences of home visits through the public health department, a school-based clinic, a homeless clinic, and the AIDS clinic.

Introduction of the class content began with a discussion of course requirements and expectations. Class content was outlined in the course syllabus. A definition of PBL as well as the process was explained to the students in the first class. Particular attention was given to the differences in PBL group work and previous nursing courses' group involvement. Students were then randomly assigned to one of four groups for the weekly and semester long PBL activities.

Two faculty acted as facilitators in the class for the groups. Although both faculty were community health clinical nurse specialists, only one had extensive training and knowledge of the PBL process. During a typical class day's first hour, a mini lecture, short video, slides, guest speaker, group or panel discussion were utilized to introduce a specific topic. Prior to the class, all students were expected to review the assigned readings and be prepared to interact in class. In the second hour of class, a real-life problem was introduced and the students worked in their respective groups to solve the problem. Resources available to the students during the class included a practitioner appropriate for the day's topic, a computer lab with Internet access located within the School of Nursing, and a mini-library consisting of current textbooks and journals from community health nursing as well as from the areas of microbiology, pathophysiology, nursing management, pediatrics, adult health, mental health, and o bstetrical nursing.

Problems distributed in class were developed from actual practice situations and based upon theoretical content. Frequently, these problems correlated with the students' own clinical experiences. Tutorial guides outlining key components and solutions for each problem were developed for the facilitators with each of the four modules. The four modules were carbon monoxide poisoning of a family, food borne illness outbreak involving 200 individuals, a bacterial infection, and a natural disaster.

One of the most well received and highly rated of the in-class problems was one on disasters. A few months prior to the start of the class, a level F4 tornado came within ten miles of the university. Having students in the class who had cared for or knew someone injured in the disaster highlighted the actual and potential of the disaster problem. Many students expressed appreciation for this module and said that they knew that this was a real problem they would encounter. This module, designed to be done in one class, could be expanded to become a weeklong PBL activity. A summary of this class problem is listed below.

You are an emergency nurse in a hospital in a large medical center. Security reports that a tornado warning has been issued for the area. The Emergency Medical Service phone rings and you receive a report that there are multiple injuries reported from a tornado on the ground ten miles from the hospital. Twenty minutes later, the phone rings and your emergency department is to receive the following patients within 3-5 minutes.

Using the aforementioned information, five patient scenarios were then presented. In a ten-minute period, each group must prioritize the patients. Upon reconvening, an emergency nurse acting as the one of the facilitators had each group justify their decisions. In the second half of the problem, information on the two more serious patients was given as to their respective presentation in an emergency department. Within each group, students evaluated prior decisions and determined what additional questions, data, resources, and interventions would be required. Thirty minutes was allowed for the evaluation and upon completion, each group discussed and defended their position on the problem.

Sixty-five percent of the course was derived from group PBL activities. Fifteen percent of this was attributed to in-class problems and the remaining fifty percent involved a semester-long community project. The rest of the course assignments were individual student work and included clinical journals (10%), a natural history paper on a selected problem or disease (10%), and a comprehensive exam (15%). Due to the commitment by faculty and the knowledge that these particular students would soon be practicing in settings involving interdisciplinary teams, PBL comprised the majority of the course. Indeed with PBL, each student has the opportunity to be an active member of a team and work toward a common goal.

Assessment of student work was performed using an adaptation of the tool developed by Amos and White, 1998 which evaluated the student's quality of work, content/process knowledge, application to practice, group participation, and ability to abide by the rules of trust. Quality of work was determined by the organization, timeliness and accuracy of submitted work. Content/Process involved the balancing of the learning process through oral and written assignments. Application to practice involved demonstrating the ability to synthesize and apply learning to other practice areas and populations. Facilitators evaluated the group participation paying particular attention to a student's activities, critiques discussion and application of learning. The Rules of Trust involved the peer-evaluation component. An example of the rules of trust is displayed in Table 1. Each student evaluated every other member in the group. The peer evaluation comprised one fifth of the evaluation on each problem. The group members decid ed what criteria were to be evaluated for the rules of trust.

SEMESTER-LONG PBL PROJECT

As previously noted, a large component of the PBL evaluation involved a community- management project. For this project, students were assigned to a different group. Similar to the weekly groups, each group consisted of four to five students. These groups functioned separately and independently of the weekly PBL groups. The emphasis of the PBL Project was to enhance students' creativity and critical thinking. Each group was assigned a real-life problem in a selected community.

The course coordinator along with a community representative confirmed the problem to be addressed prior to the onset of the project. For the purpose of the project, a community was defined as "an open social system characterized by people in a place over time who have common goals" (Smith & Maurer, 2000, p. 342). The communities selected for the projects were a small city, a school-based clinic, a homeless clinic, a multi-cultural elementary school, a diagnostic center in a major hospital, and an outpatient/surgical center for children. The specific goals of the project were to:

* Describe the problems of the community

* Identify the potential and actual resources available to the identified patient (s) in the problem or others seeking care in the community.

* Develop a systematic method for identifying the health, environmental, management and social issues in the community.

* Analyze morbidity, demographic, and healthcare issues that are affecting the community and its members.

* Describe the community's management style including organizational structure, funding sources, and budget.

* Analyze the gaps and barriers in resources.

* Recommend solutions for the identified problems.

Ample clinical time was allotted for the student groups to gather the needed information through computer and library searches, Centers for Disease Control (CDC) and Vital Statistics data, community visits, and community officials interviews. Over the next several weeks, group members took responsibility for completing identified tasks. Facilitators met weekly with the groups to assess progress and offer feedback. We approached this activity from the philosophy of Philip C. Schlechty who stated." learning involves action; most of what students learn comes from what they do which includes listening, creating and talking; regardless of the mode or style of learning, it is what students do and the meaning they give to it that determines what they learn." (Schlechty, 1997, 42).

At the end of the semester, each group presented their findings in a written formal paper as well as a professional poster. The poster was presented with an oral defense to peers, faculty, other nurses, an external nursing consultant, and invited guests from the specific communities. The findings presented on the poster and in the paper included justification on the basis of research which permitted the course faculty to evaluate the students' understanding of the parameters of the designed problem. The poster presentation allowed the students to discuss and display their data and interpretations one-on-one in a small group atmosphere. (Former, Bisson and Loretz, 1998)

RESULTS

Evaluation guidelines for individual work of the natural history paper, peer review of the paper, pre and post essay exams, clinical requirements, and facilitator evaluation of student and clinical journals were given to each student. All these activities were graded. Quantitative evaluations were obtained through a pre and post-content essay exam given to all students and the Health Education Systems, Inc. (HESI) exam (Lauchner & Britt, 1998) which is a 160 item comprehensive multiple choice test with critical thinking items similar to those on the national licensure exam. The HESI test is designed so that the test taker must use critical thinking skills to answer the questions correctly. This exam was given at the beginning and end of the term. Although this exam was graded, the score was not computed in the grade for the course. It was expected that increased test scores would be the product of the PBL course given the strong emphasis placed on critical thinking skills.

The HESI revealed increases in all areas (Table II). There were significant increases in the probability scores from the pretest to the posttest (p =. 001). Areas that increased of particular interest to this included: analysis from 88.75 to 99; reduce risk potential from 83.70 to 96.35. This may be due to the strong clinical application component of this course in areas of public health, home care, homelessness and school health.

Students completed a Student Attitudes and Activities Assessment (SAAA) developed and provided by an external assessor. This tool examined student's attitudes toward a variety of teaching and learning strategies. A Likert scale from 1-5, strongly agree to strongly disagree was used to grade responses. Students participating from the School of Nursing showed little attitude differences between the onset and completion of the course for the majority of attitude items. This observation may be attributed to the fact that these groups of students were all registered nurses with a variety of experiences and previous exposure to group work and interdisciplinary collaboration. Of those students who did not have work experience, all had medical mission experience in Africa and/or Venezuela. Regarding specific items from the SAAA, for item 9, "I feel comfortable working and participating in small groups"; there was no change. Students were comfortable with group/team work at the beginning and end of the course. This f inding is not surprising considering that these nursing students have had many opportunities in the nursing curriculum and in life or work experiences to participate in group/teamwork.

Also, perhaps because of their maturity level and work experience, students did not expect that the teacher would be primarily responsible for their learning in the beginning or at the end of the semester. An additional item of interest was item 11; "I feel the instruction of this course is similar to other classes that I have taken at Samford." For this item, students' attitudes changed. At the end of the semester, students indicated that this PBL course was not similar to other classes taken at Samford. Even though group work had been a part of previous courses, it was had not been for an entire semester and the format of the other courses were still primarily lecture/discussion.

An additional tool used to assess students in this class was one that the nursing PBL team developed. This tool was a twenty-three item pre-PBL survey, which was completed in the spring by all the students who were scheduled to take Community Health Nursing the following fall. The survey assessed demographic information, perception of group work, styles of learning, and preferred method of evaluation. At the end of the fall term, the students were given the survey again to determine if their perceptions, styles of learning or preferences. Overall the descriptive findings were similar at the beginning and end of the semester for reasons previously stated. There was a significant difference (p =. 0067) in the pre and post results for item 11, "I find the teaching/learning method of case studies to be not helpful to extremely helpful." In the pre-assessment, only 63.2% thought case studies were helpful to extremely helpful. In post assessment, 95% found case studies to be helpful to extremely helpful. One stude nt (5%) felt case studies were only somewhat helpful.

Student evaluation of PBL was obtained through qualitative data at mid-semester, after the semester long project and at the end of the course. Qualitative data was obtained with the open-ended question, "What did you learn from this PBL experience?" These comments were reviewed for key words such as group/teamwork, critical thinking, leadership, and application of learning, and categorized. These findings are reported as frequencies in Figure 1. Comparisons were made in mid semester, project and end of course responses. The number of varied comments increased as did the number of categories. All students at the end of the term responded that they had learned more about team/group work.

The assignment of the journaling gave the students the opportunity to describe patient problems, use critical thinking, and collaborative interventions throughout the semester. The faculty in the course also did journaling. The journals validated the PBL process through the students ability to problem solve in the clinical setting. Creedy and Hand (1994) recommend that PBL faculty keep a reflective journal after PBL sessions. This allowed the faculty to explore their evolving personal reflections as the course progressed.

DISCUSSION

As nursing educators with a variety of teaching experiences with students from varied backgrounds and educational level, we considered ourselves to be innovative and creative educators. Yet, no other active strategy has produced the results we received with PBL. There was nothing to stifle the students' thoughts, ideas, or comments from freely flowing into the learning environment.

Students, that were passive in previous courses, began talking, volunteering to do Internet searches, asking thoughtful questions, leading small group discussions, and reporting valuable findings on a weekly basis. This finding was most evident in the semester long project. In previous courses, group work had been a part of the course but never with the same people working on the same problem for this long an interval. By being a part of the same group for three months, the students developed much more than problem- solving skills. In their own words, they related: "that a diverse group can complete a project we can all be proud of by using teamwork;" "to delegate and accept responsibility;" "to work as a team."

Others internalized the process reflecting more of a self-assessment during this process indicating personal and professional growth. Comments included: "differences can be resolved in a civil manner;" "it is important to communicate on a personal level." "I was challenged to think in class;" and "it is important to be open-minded and willing to admit I may be wrong."

The placement of this PBL experience in the final semester of the senior year was positive in a way that was not foreseen. The nursing curriculum was a natural fit to PBL. For one, there was no shortage of interesting, real-life problems. Second, the students had a base of knowledge from which to draw from and apply PBL principles in an entirely new setting. Students also sought out professional meetings such as the National Conference on Undergraduate Research (NCUR) and were accepted to present. Many students were encouraged and inspired to pursue graduate study based upon the evaluations of these PBL projects. Of the 24 students, eight students enrolled in graduate study and five more have been accepted.

The assertion to be proven in redesigning this course to PBL was that this new methodology could improve and achieve learning outcomes. Findings support PBL, as a strategy that allows students to achieve the learning outcomes of this course. As the final senior level course for baccalaureate students, the course objectives were designed to prepare them to go out into the real world and practice as professional nurses. The Essentials of Baccalaureate Education (1998) recommend that graduates have these core competencies: critical thinking, communication, assessment, and technical skills. Core knowledge is expected in health promotion, risk reduction, disease prevention and management, ethics, diversity, global health care, heath care systems and technology.

Results as reported in Table II and Figure I support that PBL improved learning outcomes and achieved these core competencies. In addition, the students' prior education, life, and work experiences combined with the opportunity to apply this knowledge through weekly PBL activities, the PBL semester long project and the clinical experiences also supported that this methodology improved learning outcomes.

LITERATURE CITED

Alexander, J.G., Baldwin, M.S. & McDaniel, G.S. (1998) Achieving Educational Outcomes in Community Health Nursing through PBL. PBL Insight. 1. (2) 5-7.

America Association of Colleges of Nursing (1998). The Essentials of Baccalaureate Education for Professional Nursing Practice. Washington, DC: Author.

Amos, E. & White, M.J. (1998) Teaching Tools: Problem-Based Learning. Nurse Educator, 23 (2), 11-14.

Barrows, H. (1985) Designing a Problem Based Curriculum for the Pre-Clinical years. New York: Springer Publishing Company.

Barrows, H. & Tamblyn R. (1980) Problem-based learning: An approach to medical education. Springer, New York.

Bruer, J.T. (1993) Schools for thought: a science of learning in the classroom. Cambridge: MIT Press.

Bruning, R.H., Schraw, G.J. & Roning, R.R. (1995). Cognitive psychology and instruction (2nd ed.). Englewood Cliffs: Prentice Hall.

Butterfield, P.G. (1990) Thinking upstream: Nurturing a conceptual understanding of the societal context of health behavior. Advances in Nursing Science, 12 (2), 1-8.

Creedy, D. & Hand, B. (1994) The implementation of problem-based learning: changing pedagogy in nurse education. Journal of Advanced Nursing Practice, 20, 696-702.

Fortner, C.R., Bison, M. and Loretz, C.A. (1998) Using posters in case studies: the scientific poster as a teaching tool. http://ublib.buffalo.edu/library/projects/cases/posters.htm: UB Case Studies in Science.

Frost, M. (1996) An analysis of the scope and value of problem-based learning in the education of health care professionals. Journal of Advanced Nursing Practice 24: 1047-1053.

Gijselaers, W.H. (1996) Connecting problem-based practices with educational theory. In Wilkerson & W.H. Gijselaers (Eds.), Bringing problem-based learning to higher education: theory and practice. San Francisco: Jossey-Bass.

Heliker, D. (1994) Meeting the challenge of the curriculum revolution: Problem-based learning in nursing education. Journal of Nursing Education 33, (1), 45-47.

Lauchner, K.A. & Britt, R.B. (1998) External curriculum evaluation: A model for comparative analysis and predictability using the HESI exit exam. (abstract). Point Blank, Texas: Health Education System Schlechty, P.C. (1997) Inventing Better Schools. An action plan for educational reform. Josey-Bass Inc. San Francisco, California 

Smith, C.M. & Maurer, F. A. (2000) Community Health Nursing Theory and Practice W.B. Saunders: Philadelphia, PA.

                     Health Education System, Inc.
                       HESI Results Comparison of
                     Pre and Post Class Probability
                     Scores In Selected Categories
                 Relevant to Community Health Nursing
Category                     Pre Class Score  Post Class Score
Assessment                        89.67            95.45
Analysis                          88.75            99.00
Planning                          88.82            99.00
Implementation                    83.54            94.45
Physiologic Integrity             88.71            99.00
Medical/Surgical                  91.18            99.00
Pediatric                         87.10            99.00
Gerontological                    77.10            89.90
Reduce Risk Potential             83.70            96.35
Basic Care and/Comfort            76.92            99.00
Pharmacologic/Parental Tx         94.63            99.00
Physiologic Adaptation            90.31            99.00
Coping and Adaptation             80.41            89.95
Prevention and Detection of
Disease                           81.24            97.13

[Graph omitted]