Efficacy of student's self-assessment.

Abstract

This study examines research on high and low achievers with a particular focus on 30 undergraduate students. Critical student achievement behaviors are reviewed using a critical reflective inquiry process. Through the use of a survey, the disparities between student and instructor ratings of student behaviors that contribute to achievement were explored. Preliminary results reveal that a significant difference exists between teacher and student ratings of academic behaviors. Self-assessment and its problematic issues are reviewed, particularly as they relate to increased student awareness and learning.

Introduction

Reflective inquiry-based science and current models of teaching and learning require that students become more active in their learning. For many students, getting acclimated to new modes of teaching while assuming more responsibility over their learning than has been traditionally expected, is not an easy task. According to Lob, Reiser, Radinsky, Edelson, Gomez and Marshall (2001), "reflective inquiry is a style of inquiry that encompasses both effective inquiry strategies (e.g., systematically collecting and interpreting information) and reflective activities (e.g., monitoring, periodically evaluating progress, and revising plans)." A critical component of the "reflective inquiry" process is self-assessment. An accurate appraisal of oneself requires the ability to critically examine oneself and appraise academic/learning strengths and weaknesses, and then use that analysis to formulate a study plan that involves both in-class and out of class behaviors (Mentkowski, 2000).

Reflective Inquiry and Self-Regulated Learning 

In educational research today, there is a significant amount of literature concerning reflective thinking and self-regulation in teaching and learning (e.g., Anzul & Ely, 1999; Curtis & Szestay, 2005; Ghaye, 2000; Glazer, Abbott, & Harris, 2004; Griffin, 2003; Newman, 1999). The topic of self-regulation has a long history of research that emphasizes the fact that students with poor self-regulation skills achieve poorly in school (Zimmerman & Martinez-Pons, 1988). Many researchers have furthered argued that many of the problems students exhibit with low achievement has more to do with difficulties with self-regulation rather than poor learning behaviors alone (Carey, Evans, Honda, Jay and Ungen, 1989; Shauble, 1990).

According to Boekaerts and Pintrich (1999), self-regulated learning is "an active, constructive process whereby learners set goals for their learning and then attempt to monitor, regulate, and control their cognition, motivation, and behavior, guided and constrained by their goals and contextual features of the environment" (p. 453). Even though the changing demands of students' challenges them to utilize reflective inquiry practices to succeed academically, many of them do know neither where to begin nor how to proceed (Kyza, Golan, Reiser, & Edelson, D.C., 2003).

Student achievement and it relationship to self-regulated learning continues to be a major area of concern. Most research on self-regulation has been conducted on individual students. However, to our knowledge, in the field of education, very little of that research has focused on looking at the ability of students to accurately identify their own self-regulation behaviors as it relates to achievement.

Self-Assessment and Student Achievement

A review of literature on self-assessment over the past 30 years indicates mixed results on the efficacy and desirability of self-report measures (Pascarella & Terenzini, 1991). Results varied depending on the traits and instruments examined. Variations in results were observed when one compares individual versus group performances on self-report instruments (e.g., Anaya, 1999; Pace, 1985; Pike, 1995, 1996). For example, in a study exploring the use of self-reporting measures, Kuh (2005) reported that self-report measures were reliable and valid when the following conditions exist:

1. Respondents information is requested;

2. Questions are phrased clearly and unambiguously;

3. Questions refer to recent activities;

4. Respondents think the questions merit a serious and thoughtful response;

5. Answering the questions does not threaten, embarrass, or violate the privacy of the respondent or encourage the respondent to answer in socially desirable, rather than truthful ways.

Other studies continue to highlight the disparities between students estimated knowledge and skills and the appraisal of their supervisors (e.g., Fjortoft, 2006; Mentkowski 2000; Hansford & Hattie, 1982). According to Dunning (2006), respondents' level of competence relative to self-reported information may have a significant impact on the accuracy of their reporting. Less competent respondents on self-report measures are frequently unaware of the disparity between their knowledge and skills and those of more competent respondents (Dunning, 2006). In an attempt to address some of the gaps in research, this study explored not only the differences in the ratings of classroom behaviors between high and low achievers, but we extended that line of research by investigating the disparities in instructor and student ratings of achievement behaviors in and out the classroom. More specifically, the purpose of this non-experimental field study was to investigate how undergraduate students in education courses self-assess their learning of course competencies as compared to their instructor ratings using the process of critical reflective inquiry.

Methods

Research Design

The assumption used in this field study was that, to some degree, all adult students in undergraduate education programs could use the process of critical reflective inquiry to identify and examine their own achievement behaviors. The primary research questions that guided this investigation were: How do students' ratings of their achievement behaviors (listening and study skills) compare to that of their instructor? Given the past research, we hypothesized that significance differences existed between student perceptions of their own achievement behaviors as compared to their instructors. More specifically, we believe that higher achieving students were more likely to rate themselves similarly to their instructor. Conversely, lower achieving students were more likely to rate themselves higher than their instructor, resulting in a higher degree of disparity with instructor ratings.

Participants and Survey Instrument

Thirty students participated in the study: 23 Early Childhood/Special Education majors and seven Elementary majors in the School of Education. This purposive sample of students represented the number who volunteered to participate in this study. The survey instrument was distributed and collected by the co-investigator and a second faculty member, who taught one of the education courses in the Education department. The surveys were administered in groups and treatment of participants was in accordance with the ethical standards of the University. All students had already participated in reflexive practice exercises as part of the course activities, but as this was a within-group design, this prior experience would not bias the results.

Listening and Study Skills Survey (LSSS)

The Listening and Study Skills Survey (LSSS), which was a 30 item self-administered survey, was the primary data collection instrument used in the study. The LSSS was developed by the coauthor and designed to utilize a 3-point Likert rating scale (Always, Sometimes, Never). It required students to think critically about how their individual student behaviors contribute to their academic success or failure. The survey items covered 4 broad categories of student behaviors: (1) study behaviors outside of the classroom (e.g., Managing my time to meet school, job, and relaxation needs), (2) participatory behaviors in the classroom (e.g., Contributing during class by raising my hand to answer questions without being called on by the instructor), (3) knowledge of one's own learning style (e.g., Using visual aids to help learn (e.g., notes, pictures,, graph), and (4) emotional connectivity in the classroom (e.g., Avoid emotion-packed words, phrases, or cliches)

Data Collection

To examine the classroom behaviors of undergraduate students, the LSSS was disseminated to the students in 3 education courses. Surveys were handed out approximately l0 minutes prior to courses beginning for the day. A brief overview of the study was explained to the students and the class was collectively asked to participate. All students were informed that their participation in this study was completely voluntary, and their responses would contribute to the study of the critical self-reflection of their listening and study skill behaviors that were linked to academic achievement. The students were also told that survey responses would not be anonymous because the investigators were interested in determining whether there was a disparity in instructor ratings versus student ratings of academic behaviors.

Prior to beginning the survey, the students were assured that their course grades would not be impacted by their responses marked on the survey. All the participants completed the survey within 8 minutes and returned the completed survey to the instructor teaching the course. All the names of the students who completed the survey were compiled on a list. All returned questionnaires were independently reviewed by the survey administrators for completeness. It had been determined a priori, that those questionnaires containing items that were not completed would not be included in the analysis. One week after independent review, the completed surveys underwent a group review by the administrators of the survey. The mean percentage of agreement in the independent review was 100 percent. Since all the questionnaires were unanimously agreed upon as complete, all the surveys were included in the analysis. After the completeness of the student surveys were determined, both administrators of the survey completed a similar survey for each student who participated in the study to examine potential disparities in student teacher rating of student's academic performance.

Data Analysis

To examine to disparities between instructors and student ratings, a correlation coefficient R, using SPSS, was performed to determine the index of the magnitude of the association between the variables addressed in the research question. A follow-up item analysis was conducted by hand searching the survey items to further determine the specific discrepant survey items.

Survey Return Rate

Of the 30 survey instruments distributed, 30 (100%) were returned. Of the students who participated, 29 were female and 1 was male. Seventeen of the participants were coded as high achievers (3.2 GPA or higher) and 13 were coded as low achievers (GPA under 3.2). All participants were in the Education program at Bowie State University: one was in her sophomore year, 20 were in their junior year, and 9 were in their senior year. No freshman participated in the study. Analysis of the responses based on education status did not reveal significant differences among participants. Thus, the results are presented as an overall group.

Results

A Pearson r was conducted to measure the relationship between instructor and student ratings in regards to student achievement behaviors. Pearson's correlation between instructor ratings (M=2.28, S=.692) and student ratings (M=2.35, S=.638) was r(28) = .763, p < .05. This report shows a statistically significant disparity between instructor ratings and student's ratings of student achievement behaviors. Specific areas that showed the greatest disparity in ratings included frequency of questions asked/answered, in class listening and study skills. Perceptual distortions in the use of critical student behaviors was also noted by low achieving students' tendency to rate themselves higher than the instructor on survey items such as "contribute during class by raising my hand to answer questions without being called on by the instructor" or "taking lecture notes that are complete, clear, and useful" whereas on several occasions, higher achieving students tended to rate themselves lower than the instructor rated them.

Discussion

The purpose of this study was to explore the efficacy of student ratings. Results of this study found low correlations between students' self assessment and actual performance and are consistent with previous research (Fjortoft, 2006). The results are considered important for all educators. As the above preliminary data points to the need to help students develop self-assessment skills, Fjortoft (2006) argues that to facilitate the process and be effective, any self-assessment must be accompanied by verification to support the self-assessments. The findings of this study showing higher student perception than teachers' perceptions of their academic achievement behaviors further supports Fjortoft's (2006) contention that the use of self-assessment instruments as a method for reflection alone cannot and does not make self-assessment accurate and meaningful to students.

These initial findings call for a more vigorous and detailed study of critical student achievement behaviors both in and outside of the classroom and motivational issues tied to self-worth. The significant differences found between student and teacher ratings suggest that before efforts to enhance student academic performance can be addressed, attention must be given to assisting them with the perceptual distortions they posses of their own learning behaviors. Teacher awareness of the importance of teaching self-assessment and academic skills such as note taking, material skimming, and good listening habits are key to providing low achieving students the skills necessary for higher achievement.

While the result of this study provides critical information for educators, especially those working with the students in the sample of this investigation, it is important to mention two major limitations of the study. One of the limitations of this study is the sample size. Quantity and quality of insights into the efficacy of self-ratings and reflective practice is not necessarily a function of the number of participants in a study. In a similar study, Glazer, Abbott and Harris (2004) based their study on data gathered from five elementary school teachers who met to investigate the nature of professional reflection. In spite of the small group size, Glazer, Abbott and Harris (2004) concluded that because of the value of the process and results to the teachers involved, the study was extremely valuable to furthering the education field. The same claims can be made for this study. A second limitation was the nature of instructor ratings. The focus of this study was to investigate the accuracy of student ratings, consequently, no attention was given to instructor ratings. Perhaps, future studies should look at not only verifying student's self assessment of their learning, but also employ a cross verification method for the instructor ratings of the student too.

References

Anaya, G (1999). College impact on student learning: Comparing the use of self-report gains, standard test scores, and college grades. Research in Higher Education, 40, 499-526.

Anzul, M. & Ely, M. (1999). Halls of mirrors: the introduction of the reflective mode, Language Arts, 65(7), 20-27.

Boekaerts, M., Pintrich,. P.R. (Eds) (1999). Handbook of Self-Regulation. Academic Press Publishing.

Brookfield, S. (1995). Becoming a critically reflective teacher. San Francisco: Jossey Bass.

Carey, S., R., Evans, M., Honda, E., Jay, & Unger, C. (1989). An experiment is when you try it and see if it works: A study of grade 7 students' understanding of the construction of scientific knowledge. International Journal of Science Education, 11(5), 514-529.

Curtis, A. & Szestay, M. (2005). The impact of teacher knowledge seminars: unpacking reflective practice. Teaching English as a Second or Foreign Language, 9(2).

Dunnning (2006). An Essay Not Knowing Thyself. Retrieved August 10, 2006 from http://outofthejungle.blogspot.com/2006/05/ know-thyself-students-selfassessment.html

Fjortoft, N. (2006). Self assessment in pharmacy education. American Journal of Pharmaceutical Education, 70(3), 64.

Ghaye, T. (2000). Into the reflective mode: bridging the stagnant moat (editorial). Reflective Practice, 1(1), 5-9.

Glazer, C., Abbott, L. & Harris J. (2004). A teacher-developed process for collaborative professional reflection. Reflective Practice, 5(3), pp. 33-46.

Griffin, M. L. (2003). Using Critical Incidents to Promote and Assess Reflective Thinking in Preservice Teachers. Reflective Practice, 4(2), 207-220.

Hansford & Hattie (1982). The relationship between self and achievement/performance measures. Rev Ed Res, 52:123-42.

Johnson, D.W. , Johnson, R.T., & Smith, K.A. (1991). Active learning: Cooperation In the Classroom. Edina, Mn: Interaction Book.

Kuh, G.V. (2005). Using student and alumni surveys for accountability in higher education. In J.C. Burke Edit, The Many Faces of Accountability. San Fransciso: Jossey-Bass.

Kyza, E. A., Golan, R., Reisser, B.J., & Edelson, D.C. (2003, April) Reflective Inquiry: Enabling group self-regulation in inquiry-based science using the progress portfolio tool. Paper presented at the Annual Meeting of the American Educational Research Association: Chicago, IL.

Lob, B., Reiser, B.J., Radinsky, J., Edelson, B.J., Gomez, L., & Marshall, S. (2001). Developiong reflective inquiry practices: A case study of software the teacher and student. In K. Crowley, C. Schunn, & T. Okada (Eds). Designing for Science: Implications from Everyday Classroom & Professional Settings. Mahwah, N J: Erlbaum.

Mentkowski, M. (2000). Learning That Lasts. San Francisco, CA: Jossey-Bass.

Newman, S. (1999). Constructing and critiquing reflective practice. Educational Action Research Journal, 7(1), 145-161.

Pace, C.R. (1985).The credibility of student self-reports. Los Angles: University of California. The Center for the Study of Evaluation, Graduate School of Education.

Pascarella, E.T. & Terenzini, P.T.(1991). How college affects students. San Francisco: JosseyBass.

Pike, G.R. (1995). The relationships between self-reports of college experiences and achievement tests scores. Research in Higher Education, 36,1-22.

Pike, G.R. (1996). Limitations of using students' self-reports of academic development as proxies for traditional achievement measures. Research in Higher Education, 37,89-114.

Schmeiding, N. (1999). Reflective inquiry framework for nurse administrators. Journal of Advanced Nursing, 3, 631-639.

Shauble (1990). Belief revision in children: The role of prior knowledge and strategies for generating evidence. Journal of Experimental Child Psychology, 49, 31-57.

Streubert, H. & Carpenter, D. (1995). Qualitative research in nursing. Philadelphia: Lippincott Company.

Zimmerman & Martinez-Pons (1988). Construct Validity of a strategy model of student self-regulated learning. Journal of Educational Psychology, 80, 284-290.

Thelon Byrd, Bowie State University, MD

Rochelle Matthews-Somerville, Bowie State University, MD

Byrd, Ph.D., is a Professor and Coordinator of Graduate Special Education and Matthews-Somerville, Ph.D., is an Assistant Professor and Coordinator of the Undergraduate Early Childhood/Special Education program.