Portfolio assessment and self-directed learning.
Portfolios have been heralded as an assessment method to enhance students' self-directed learning. This pilot study was done to better understand the nature of such issues from the perspective of four physicians who had completed a showcase portfolio requirement during their graduate medical training. The physicians were interviewed 1-2 years after completing the portfolio. Results from survey data and from content analysis of semi-structured interview responses suggest that a showcase portfolio tool may be most influential on physician's skills and abilities to critically analyze their own clinical performance.
Several recent authors have discussed compelling ideas about how educators could implement and use tools such as student portfolio assignments to help develop critical thinkers and self-directed learners (Mentkowski, 1991, Mentkowski, et al., 2000; O'Sullivan et al., 2002). Within the realm of postsecondary health professions education, such assessment approaches are appealing because they are an authentic method that can be used to promote a students' self-reflective analysis of their performance within the health practice contexts in which they gain much of their clinical training (Spencer & Jordan, 1999). Another appeal is that portfolio tools can be structured to provide information that will simultaneously help educational leaders meet institutional and/or program improvement objectives and goals. There is research evidence to suggest that portfolio tools are feasible to implement in psychiatry residency training programs (O'Sullivan et al., 2002). There is also evidence that portfolios may be useful for evaluating a residents' performance as well as program improvement (Jarvis et al., 2004; McClain et al., 2004).
The intent of the study described in this paper was to initiate our understanding of whether and how completing a showcase portfolio requirement (that serves primarily as a program assessment tool rather than a student learning portfolio, per se) might influence aspects of self-directed learning for recent psychiatry resident graduates. Few studies have examined the influence of assessment methods such as portfolios on self-directed learning in residents. More generally, empirical evaluations of the impact of assessment methods on student performance have been relatively rare as well (Peterson & Einarson, 2001). To date, most of the research on self-directed learning in medical education has focused on the relative impact of elements of the curriculum on self-directed learning indicators (e.g., traditional didactic lectures vs. PBL) (Ozuah et al., 2001; Shin et al., 1993). As a whole, this research has produced mixed findings (see Schmidt, 2000).
A useful conceptual framework of self-directed learning in clinical practice has been previously described by Harvey et al. (2003) and Miflin (2000), who suggest that the essential components of self-directed learning for clinicians include having the motivation and ability to self-assess learning needs, the ability to efficiently locate and use a range of appropriate resources (e.g., literature search, mentors, etc.), and the ability to critically evaluate learning outcomes.
Study Design and Setting The current study was approved by the appropriate Institutional Review Board. The study was a small pilot project intended to generate hypotheses. Descriptive qualitative methods and computer content analysis of semi-structured interview responses were the primary methods utilized. Several authors have recently noted the potential benefits of using a qualitative approach to better understand the nature of self-directed learning (see Baveye, 2003).
The psychiatry residency program from which eligible study participants were drawn is an accredited four-year program based at a medical school in the mid-south U.S. As noted in detail elsewhere, the showcase portfolio requirement for psychiatry residents was first implemented in 2000-2001 (O'Sullivan et al., 2002). Residents are expected to select several elective and required areas of competency for their portfolio entries that reflect his or her best work across 13 necessary psychiatric skills (e.g., initial evaluation and diagnosis, treatment course, sell-directed learning, working with teams, crisis management, medical psychiatry, legal issues, presentation/teaching skills, specific treatment modalities, neuropsychiatry, professional communication, bio-psychosocial/spiritual formulation, and psychotherapy). An entry consists of actual resident work and self-evaluation within the skill area, and may be comprised of chart documentation, laboratory or radiology records, literature searches, and various other relevant data.
Sample A purposive sample of recent graduates from the psychiatry residency program was selected based on the criterion that they had completed at least one year of the portfolio assignment while in the program. Eight graduates were eligible and invited to participate via a cover letter (e-mail or postal mail); ultimately four were interviewed. Three eligible participants did not respond to the initial invitation and were unable to be reached via phone or mail follow-up attempts; one responded as interested, but was unavailable for interview during the study time frame. Of the four psychiatrists who were interviewed, two were male and two were female, all were employed primarily in community mental health clinic settings. Interviews were led by a research associate with extensive health services research experience. One interview was completed face-to-face, and three were completed via telephone. Participants were informed about the purpose of the study, they completed two survey instruments and were asked a series of semi-structured questions (described below). The survey administration and interviews lasted about 30-50 minutes.
Measures /Data Sources The psychiatrists completed two brief surveys, one titled "How do I learn" (available from: http://www.prioritymanagement.com), and another developed specifically for this project. The surveys were selected and designed to assess: 1) aspects of self-directed and lifelong learning; and 2) effects of the portfolio experience oil broad domains of self-directed learning (e.g., several items modeled after factors from the Self-Directed Learning Readiness Scale; Gugliemino, 1977). The learning survey consisted of 7 items rated on a 4-point scale ranging from strongly agree to strongly disagree. Total scores could range from 0-21 with higher scores indicating greater efficacy/commitment to self-directed learning and lifelong learning. Representative items are: I am an effective independent learner, and It is my responsibility to create learning opportunities, not my employer's. Seven survey items were also developed to better understand whether the portfolio experience improved any of the following aspects of self-directed learning: self understanding, ability to be responsible for own learning, tolerance of risk, ambiguity and complexity in learning, efforts to analyze own professional practices, ability to critically analyze own work, number of medical discussions with colleagues, and number of self-study hours.
Several semi-structured interview questions were created based on review of the literature and discussions with health professions colleagues. These interview questions basically encouraged participants to talk about their experience, in general, in completing the portfolio, the role of the portfolio experience on their work, and how they stay current with developments in their field. Immediately following the one interview that was conducted face-to-lace, a summarized transcript was typed up by the interviewer and verified by a member check with the respondent. For the remaining three interviews, conducted by telephone, the interviewer took extensive notes during the course of the conversations and was able to record respondents' comments almost completely verbatim; transcripts were typed immediately following the interviews.
Data Analysis Descriptive statistics were computed for responses to the survey items. Responses to the semi-structured interview questions were analyzed for prominent themes using the General Inquirer (GI) Computer Content Analysis Program (Stone, et al, 1966). The current version of the GI (Buvac & Stone, 2001) contains 182 pre-determined categories primarily derived from the Harvard IV-4 Psychosocial Dictionary and the Lasswell Dictionary. Briefly, the GI program analyzes samples of text by recognizing, matching and counting the frequency of occurrence of words that are included in the 182 dictionary categories. For each respondents' text sample analyzed, the proportion of words in a category (relative to the total number of words analyzed for each text sample) serves as an indicator of the themes, concerns, or values that the dictionary was designed to assess. Two large valence categories are Positive which consists of 1,915 words of a positive outlook, and Negative the largest category represented in the program with 2,291 words related to a negative outlook. The categories are not mutually exclusive in that one word may fall into several different categories. Unrecognizable words are not categorized, but are still considered in the total word count for computations of categorical proportions. Transcripts from each interview were individually analyzed by the GI, excluding the interviewer questions. The proportion of words for the valence categories (positive/negative) and the top 5 most frequently used categories were the focus of analysis.
Survey Responses Total scores on the brief "How do l learn" survey were 14, 18, 20, and 16, for each respondent, respectively (mean total score = 17; maximum score possible = 21). These scores indicate a fairly high level of self-efficacy in learning and positive attitudes toward continual learning in the workplace. Responses to survey items about the portfolio experience found that psychiatrists' endorsed strong agreement with only one of the seven items (the influence of the portfolio on their abilities to critically analyze their own work). Most responses to items were rated as somewhat agree. One psychiatrist did not respond affirmatively to any of these items and the interview responses/discussion further indicated that this individual generally was alienated towards the program.
Interview Results Results from the semi-structured interview questions found that all but one of the psychiatrists noted the importance of the portfolio experience in helping them to better reflect on and critically analyze their performance. Several representative comments related to this process include the following: 1) The portfolio approach required us to select our best work; so I became much more aware of what my day to day work looked like as 1 viewed it with an eye toward how I would select my best work. 2) I definitely see the value in doing portfolios ... There's a lot to be said for such an approach, to allow you to demonstrate your own perception of your competence ... besides who knows you better than yourself!? 3) 1 learned its better to back up my work and can explain it better to others. You know, you do kind of take things for granted and doing the portfolio makes you think about your work, you really have to know your stuff. Interestingly, the one respondent noted the importance of personality characteristics on self-directed learning: I think personality traits are an important part of your approach to learning. Another comment suggested the importance of motivation and organizational culture: Self-directed learning requires one very important thing ... and that is motivation, drive ... you also have to create a culture that supports learning.
In response to the question asking how the study participants stay current with new information and developments in the field, the psychiatrists noted the following continuing education and lifelong-learning approaches: reading the latest journals, teaching others, self-study to prepare for board exams, discussions with colleagues and others involved in residency program training activities, attending continuing medical education (CME) talks, completing CME opportunities offered through medical journals, and attending programs offered by pharmaceutical representatives/companies.
Computerized Content Analysis The total word count for each interview transcript analyzed by the GI (n = 4) was 445, 191,500 and 281 words. The GI program did not recognize or have corresponding word matches for a total of 21, 4, 14, and 10, words, in each of the respective texts/transcript analyzed. Descriptive statistics for the top five GI thematic categories used by the psychiatrists, showed the most commonly occurring words fell into the category labeled Active which contains words implying an active orientation (12.9% average usage, range = 11.5--13.9), the next most commonly used category was State Verbs which refers to verbs that describe mental or emotional states (10.6 Vo average usage, range = 7.5-14.7). The remaining 3 of the top 5 most frequently used categories were Strong which contains words imply strength (9.1%, average usage, range = 5.8 - 13.2), Enlightenment (8.5% average usage, range = 4.3-12.6) which contains words related to knowledge, insight and information concerning personal and cultural relations, and Interpretive Action Verbs (7.6% average usage, range = 6.8-8.5) which is a category of words representing verbs giving an interpretation of action (e.g., encourage). The average proportion of words in the overarching valence categories (e.g. positive/negative), found that Positive words accounted for an average of 7.6% of the total words analyzed (range = 6.1-9.4). In contrast, relatively few words from the Negative category were used (0.72% average usage, range = 0.0-1.4). The developers of the GI note it is expected that more positive words than negative would appear in text samples, and that based on their experience over the years in using this program if about 10% of the words are scored as positive, that is considered quite a lot. Future study with more extensive amounts of text for analysis would help determine the stability of these pilot findings.
This study of the first cohort of psychiatrists who had completed a showcase portfolio during their final year or two in residency training provides insights for future studies on this topic. The results suggest that the showcase portfolio experience was primarily influential on one of the key aspects comprising self-directed learning--the skill and ability to critically reflect on and assess one's own work/performance. Given these initial findings, future research in this area may benefit from the use of Ryan's two part model/questionnaire (1993) which is designed to assess a student's perceptions about the importance of self-directed learning and their abilities as self-directed learners. Other domains suggested by the respondents that seem to be important factors to consider in assessment of self-directed learning are personality factors, motivation, and the influence of the educational environment and culture.
That the psychiatrists who participated in this study answered primarily affirmative in their responses to the learning survey items suggests that they had a fairly strong sense of self-efficacy as independent learners and had opportunities to continue learning in their current positions. Interestingly, however, they did not necessarily attribute gains in self-directed learning as a direct result of the showcase portfolio experience. As a whole, most comments made by the psychiatrists consisted of positive words, and the prominent themes from the content analysis results provide a reliable, albeit limited, measure of important themes discussed in the interviews (e.g., active, state verbs, strong, enlightenment, and action). Ideally, use of the GI program for content analyses of interview text as was done in this study could be improved and made more reliable by obtaining a larger amount of text/words per person to analyze.
How best to foster skill in self-assessment is an important topic for future research and practice improvements in graduate medical education, continuing medical education, and higher education in general. Portfolio assignments may be useful to facilitate such efforts. In theory, medical students/residents who are competent self-directed learners have acquired the skills and motivation necessary to sustain life-long learning; although others have noted that there is scant empirical research literature yet that addresses this topic (Schmidt, 2000).
Buvac, V., & Stone, P. (2001). The General Inquirer User's Guide. Retrieved June 25, 2004, from http://www.wjh.harvard.edu/~inquirer/j1_0/manual/manual.html.
Baveye, P. (2003). Refocusing the self-directed learning debate. Family Medicine, 35(6), 445-446.
Cleghorn G. D., & Headrick, L. A. (1996). The PDSA cycle at the core of learning in health professions education. Joint Commission Journal on Quality Improvement, 22(3), 206-212.
Guglielmino, L. M. (1977). Development of the self-directed learning readiness scale. Unpublished Doctoral Dissertation of Education, University of Georgia, Athens, Georgia.
Harvey, B. J., Rothman, A. I., & Frecker, R. C. (2003). Effect of an undergraduate medical curriculum on students' self-directed learning. Academic Medicine, 78(12), 1259-1265.
Jarvis, R. M., O'Sullivan, P. S., McClain, T., & Clardy, J. A. (2004). Can one portfolio measure the six ACGME general competencies? Academic Psychiatry, 28(3), 190-196.
Knowles, M. S. 0975). Self-directed learning. New York: Association Press.
Mentkowski, M. W., Rogers, G., Doherty, A., Loacker, G., Hart, J. R., Rickards, W., et. al. (2000). Learning that lasts: Integrating learning, development, and performance in college and beyond. San Francisco: Jossey-Bass.
Mentkowski, M. W. (1991). Creating a context where institutional assessment yields educational improvement. Journal of General Education, 40, 255-283.
McClain, T., O'Sullivan, P. S., & Clardy, J. A. (2004). Biopsychosocial formulation: Recognizing educational shortcomings. Academic Psychiatry, 28(2), 88-94.
Miflin, B.M., Campbell C.B., & Price, D.A. (2000). A conceptual framework to guide the development of self-directed, lifelong learning in problem-based medical curricula. Medical Education, 34(4), 299-306.
O'Sullivan, P. S., Cogbill, K., McClain, C., Clardy, J., & Reckase, M. (2002).
Portfolios as a novel approach for residency evaluations. Academic Psychiatry, 26(3), 173-179. Ozuah, P.O., Curtis, J., & Stein, R. E. (2001). Impact of problem-based learning on residents' sell-directed learning. Archives of Pediatric and Adolescent Medicine, 155(6), 669-672.
Peterson, M. W., & Einarson, M. K. (2001). What are colleges doing about student assessment? Does it make a difference? The Journal of Higher Education, 72(6), 629-669.
Ryan, G. (1993). Student perceptions about self-directed learning in a professional course implementing problem-based learning. Studies in Higher Education, 18(1), 53-64.
Schmidt, H. G. (2000). Assumptions underlying sell-directed learning may be false. Medical Education, 34(4), 243-245.
Shin, J. H., Haynes, R. B., & Johnston, M. E. (1993). Effect of problem-based, self-directed undergraduate education on life-long learning. Canadian Medical Association Journal, 148(6), 969-976.
Spencer, J. A., & Jordan, R. K. (1999). Learner centred approaches in medical education. British Medical Journal, 318 (7193), 1280-1283.
Stone, P. J., Dumphy, D. C. Smith, M. S., Ogilvie, D. M., & associates. (1966). The General Inquirer: A Computer Approach to Content Analysis. Cambridge, MA: MIT Press.
Carol R. Thrush, University of Arkansas for Medical Sciences Patricia S. O'Sullivan, University of Arkansas for Medical Sciences James A. Clardy, University of Arkansas for Medical Sciences Jim Vander Putten, University of Arkansas at Little Rock Carol Kamin, University of Colorado
Thrush, M.A. is a Research Associate in Psychiatry and doctoral student in Higher Education, O'Sullivan, Ed.D. is a Professor in the Office of Educational Development, Clardy, M.D. is Associate Dean for Graduate Medical Education, Vander Putten, Ph.D. is an Associate Professor of Higher Education, and Kamin, Ed.D. is an Associate Professor of Pediatrics.
|Printer friendly Cite/link Email Feedback|
|Publication:||Academic Exchange Quarterly|
|Date:||Sep 22, 2005|
|Previous Article:||Teachers' efficacy in preparation and retention.|
|Next Article:||Service learning and student academic success.|