Peer evaluation as an active learning technique.

An action research was carried out during the fall 2003 term. Nineteen students in an Applied Research in Education course at Fayetteville State University, North Carolina were participants in the study to determine the impact of using peers in the evaluation of a partial research paper. The answers to three questions were sought: 1) To what extent is peer feedback meaningful and effective: 2) What lessons (if any) do the peer-evaluators learn from the activity: and 3) To what extent does the peer evaluation process result in better research papers for student researchers and peer evaluators? Results showed that all of the student researchers agreed that the feedback was helpful, constructive, clear and understandable. Further, results indicated that the student researchers realized that they need to include more substantial information in the review of the literature section and that the activity was helpful in the final paper revision process. Overall, the quality of the papers from this class was significantly higher than papers collected from previous classes.

**********

In my introductory research course the focus is more on getting my students to learn the process than getting them to master the process. Learning the terminology, being competent in statistical computations, understanding how to analyze research and other basic components of educational research are essential knowledge for the student.

Evaluation is a crucial part of the learning process (Barrett, 1986). Consequently, feedback and evaluation on a student's progress are important both to the student and to the faculty. Students need information on their progress to make improvements in their work. Although a faculty evaluation of student work is perhaps the most common type of evaluation, the use of peers in the evaluation process may be just as effective in their academic growth.

Previous evaluations of students' work involved me, as the instructor, in reviewing drafts of papers, inserting comments in red ink, and returning the papers to the students. In some instances, students expressed how the editing of their papers was intimidating and damaging to their self-esteem. Discovering another way to effectively evaluate student research while at the same time making the evaluation process a learning activity would eliminate the problems associated with editing, such as the loss in self-esteem and increase the involvement of the class in the process.

Peer Evaluation

According to Boyd (1989), evaluation can be used to "provide constructive criticism and suggestions to improve weak areas and amplify strengths." (p.2). Two types of evaluation exist--formative and summative. Formative evaluation involves giving feedback that is focused on changing processes as they are happening. On the other hand, summative evaluation occurs at the end of the process and focuses on judging the quality of the outcome (Boulmetis and Dutwin, 2000).

Using peers has been found to be an effective addition to the evaluation process. Formative peer evaluations allow peers to work collaboratively to assess each others' work and assist one another in efforts to strengthen research. Working with one's peers allows interchange of ideas and methodologies resulting in a more refined product (Powell, 1992; Sapin-Piane, 1993). Studies by Saavedra and Kwun (1993) found that "on the whole, both field and laboratory studies indicate that peer assessment is a valid and reliable evaluation procedure." (p. 450)

Lisk (2000) argued that this process of cooperative learning must include a number of essential conditions to be successful in learning environments. The conditions include: (a) a clear set of learning objectives that are accepted by all students, (b) positive interdependence, (c) positive social interaction behavior and attitudes, and (d) individual accountability.

Portfolio assessment, self-assessment and peer-review are forms of assessment that encourage students to engage continuously and foster a deeper approach to learning. Key elements of these approaches are reflection, feedback, and integration of learning and assessment (Dochy and McDowell, 1997).

Peer Evaluation of Research Papers

There are several issues that are associated with evaluating beginning research students. Beginning research students are not only required to learn about the process of "doing" but are also at the same time applying the process to a real classroom situation.

One issue is the wide range of research skill that students bring to the class. A graduate education program generally require a research class as a core course in the curriculum and is viewed as a course which will provide a foundation for future research projects which students will have to complete in their programs. While some students will have completed a research paper or project in their undergraduate programs, for most students the graduate course will likely be their first exposure to the research process.

Another issue is the number of new concepts and the amount of information that is required in a research course. There is an ardent amount of information for the beginning students to understand in order to "do" research. For example, basic research terminology terms (e.g., independent and dependent variables, internal and external validity, hypothesis, types of research and types of experimental design) which represent basic concepts which must be understood by beginning researchers.

Lastly, students are faced with a dilemma when attempting to conduct research in their own classrooms. This aspect of applying research skills acquired in the classroom is vital to "learning how to conduct research". However, if a student chooses to use current classroom test data (i.e., pre and post test scores) as a measuring instrument, the student may need to "create" post-test scores if the college course ends before the actual post-test scores are available.

To address these issues and improve the instructional process, I sought to examine the impact of using peers in the evaluation of a partial research paper. Specifically, I wanted to answer the following questions:

1. To what extent is peer feedback meaningful and effective?

2. What lessons (if any) do the peer-evaluators learn from the activity?

3. To what extent does the peer evaluation process result in better research papers for student researchers and peer evaluators?

In this study, the students evaluated only the first part of their classmates research paper. This "partial" section inluded the Introduction, Statement of the Problem, Review of the Literature, Statement of the Hypothesis and the Methods section. The term "peer evaluation" is defined as "a participation of colleagues in the development and assessment of activities" (Benshoff, 1988). Colleagues can help each other by reviewing each other's work, and sometimes, simply sitting and talking with each other about their studies is meaningful. In this way, critical and supportive feedback is emphasized while evaluation is de-emphasized.

Methodology

This project involved the use of one graduate educational research class. The subjects in the course were nineteen (19) teachers who, when asked to rate their skills at conducting research, showed a range of scores from 3 to 10 with 6 being the median score and 5 being the mode score (see Table l). Most students expressed that they knew there was much more to learn about conducting research.

An educational research text by Gay and Airasian (2000) served as a guide for the instruments designed for this study. Prior to the beginning of the activity, rules for completing the peer evaluation forms were shared and discussed with the class.

Each student was given forms to share with two peers of his/her choice in the evaluation of the research paper. Each student provided a copy of the partial research paper to the two self-selected classmates to review and submit suggestions and/or comments about the paper. Each reviewer signed the review sheet to indicate who had reviewed the research paper for the researcher. In this way, the researcher could ask for clarification from the reviewer about suggested revisions prior to making final revisions to the paper.

The evaluation form was divided into sections that corresponded to the portions of the paper that were to have been completed at the midpoint of the course. Specific questions about the problem statement, review of related literature, hypothesis and descriptions of the participants were included on the review form. For example, a question about the "Review of the Related Literature" section asked the following: Is the review comprehensive? Although, the peer consultant had not reviewed the information for the topic, it was expected that the written information provided by the student researcher would be comprehensive enough for the reviewer to understand the topic. Since each student in the course had completed similar research papers each reviewer was knowledgeable about the type of information that was to be included in each section of the paper.

A column on the evaluation form was included for the peer reviewer to make comments relative to each question. After the forms were returned, the student researcher reviewed the comments and talked with each reviewer concerning comments needing clarifications. The student researcher then composed a paragraph about the lessons learned from the activity and rated the activity on a 5-point Likert scale (1=strongly disagree to 5=strongly agree) on four statements relative to the activity. Lastly, a questionnaire was developed for the peer evaluators to make comments about the activity.

This learning activity was administered during the mid-term examination week, approximately halfway through the semester. At the end of the activity, each student submitted the original research paper with peer review sheets and the student researcher form attached. Each student also submitted the questionnaire that evaluated the activity.

Findings

Meaningful Peer Feedback

To address this area, the student researchers were asked to respond to three questions:

1. Was the feedback helpful?

2. Was the feedback constructive?

3. Was the feedback clear and understandable?

All of the student researchers agreed that the feedback was helpful, constructive, and clear and understandable. They felt most strongly about the constructiveness of the feedback (84% strongly agreed) and least strongly about how clear and understandable the feedback was (only 58% strongly agreed). (See Table 2)

In addition to the above, the peer evaluators were asked if they were able to give constructive feedback and whether or not they felt that their feedback should be useful in helping the student researcher revise his/her paper. All of the peer evaluators responded positively to both questions. Although 100% of them strongly agreed that they gave constructive feedback, only 79% strongly agreed that the student researcher should be able to use their feedback to revise the research paper. (See Table 3)

Based on the responses of the student researchers and peer reviewers, the peer evaluation activity was meaningful and effective in this case. The student researchers found the feedback to be constructive and clear and therefore useful and the peer reviewers were confident that their feedback was constructive and helpful to the student researcher.

Peer Evaluator Lessons

The peer reviewers were asked if they understood their role as evaluators and 100% of them agreed they they did (68% strongly agreed). However, the responses of the student researchers who also served as peer reviewers were more illuminating. The comments can be categorized into the areas of "content" and "process". Statements about the content dealt with the research topic and the material written to support the stated hypothesis. Statements about the process dealt with the evaluation of the activity.

Example statements of "content" were as follows:

* I must work harder to cover every aspect of this paper so that the reader will not wonder about any validity of my hypothesis.

* I need to analyze and critique more studies for my paper.

* I need to define more words for the reader.

* It's strange how some things make perfect sense to you, the author, but aren't quite clear to another person reading the article.

* I'm not sure I agree with some of the suggestions from my reviewers.

* I learned a lot about other topics, like retention and the use of the Lightspan program.

Example statements of "process" were as follows:

* After I received my peer evaluations, the evaluators suggested I change the same things I was already considering changing.

* It's hard to look at your own paper objectively.

* I enjoyed this more than I thought I would. I was able to see that I actually knew what I was doing and that I was getting my points across.

* I am learning that we are all struggling together. I do need the critiques of others because if 2 people see that something is missing, I know it is not there.

* The job of critiquing is not easy.

* I was able to view the other students' papers from a position of critic that increased my level of researcher adeptness.

* From looking at and reviewing other research papers, I've learned a lot about the process and how things should flow.

It is clear from the comments that the students learned things about the papers as well as about the research process. Some students realized that clarity is extremely important but not always easy to achieve. Some students also found that the review process helped them do a better job as researchers since they gained a better understanding of how their papers would be reviewed.

Impact of the Peer Evaluation Activity

All of the students indicated that they intended to use the feedback to revise their papers and that they felt that the activity helped them become better researchers.

Overall the students in this course felt that peer evaluations were helpful, meaningful and effective. I noticed that many of the students were very cautious of the comments that they made on their peers' papers. Many asked me if it was all right to mark on the paper and to write certain comments. I felt that these comments stemmed from their own fear of receiving written comments on their papers. They were likewise very apologetic to their peers when clarifying their written comments.

Similarly, the students as a whole were still "learning the process" of conducting research and welcomed comments from someone other than the instructor. They viewed the peer evaluator as someone who knew the process as they knew it. It seemed to help to have another set of eyes to view their work. The ensuing discussion between the student researcher and peer evaluator also allowed the researcher to clarify any content that was unclear or confusing to the peer reviewer.

The peer evaluators felt that this activity was helpful in helping them in the final revisions for their papers. They felt that the activity was a good way to evaluate their understanding of the information that they had learned up to that point in the semester. Likewise, the students felt that it allowed them to be "facilitators" to others learning the process of conducting research.

In general, this peer evaluation activity proved to be very effective as another way to evaluate student research papers. It allowed the students to make corrections and revisions that were from their peers rather than the instructor. It gave them an opportunity to defend their research design and statements that they had written. It also allowed the instructor an opportunity to evaluate their skills as evaluators. The student researchers stated that the activity would make them better researchers.

Overall, the quality of papers from this class was significantly higher than papers collected from previous classes. This was particularly true of the review of literature section. The peer evaluation activity helped students realize that they has to include more substantial information in this section. The selection of information for this section was more representative of current information in the field and was organized in a more logical manner than in the papers presented before the peer evaluations were initiated.

In terms of the editing process and whether it continued to damage their self-esteem, this peer evaluation process helped to remove the stigma associated with the instructor editing their papers. The peer evaluation process allowed the students to experience the editing process by their peers as well as perform the editing of their peer's paper. In some strange way, it seemed to bring meaning to the value of "editing marks". As one student commented, "I am learning that we are all struggling together". As a researcher, I too am struggling to learn better ways of helping student researchers become better in the process of "doing research".

References

Barrett, J. (1986) The evaluation of teachers. (ERIC Digest No. 12). Washington, D.C.: ERIC Clearinghouse on Teacher Education. (ERIC Document Reproduction Service No. ED278657)

Benshoff, J. (1988). The effects of the Structured Peer Supervision Model on overall supervised counseling effectiveness ratings of advanced counselors in training. Dissertation Abstracts International, 49, no 11A. Accession No: AAG8902828

Boulmetis, J. and Dutwin, P. (2000). The ABCs of evaluation. San Francisco: Jossey-Bass

Boyd, R. T. C.(1989). Improving teacher evaluations. (ERIC Digest No. 111), American Institutes for Research Washington D.C.: ERIC Clearinghouse on Tests Measurement and Evaluation. (ERIC Document Reproduction Service No. ED315431)

Darling-Hammond, L., Wise, A. E., & Pease, S. R. (1983). Teacher evaluations in an organizational context: A review of literature. Review of Educational Research, 53(3), 285-328.

Dochy, F.J.R.C. and McDowell, L. (1997). Assessment as a tool for learning. Studies in Educational Evaluation, 23, 279-298.

Gay, L. and Airasian, P.W. (2002). Educational research: Competencies for analysis and applications. (6th edition) Upper Saddle River, NJ: Prentice Hall.

Keig, L. and Waggoner, M. D. (1993). Collaborative peer review. The role of faculty in improving college teaching. (ERIC Digest 071). George Washington Univ. Washington DC. School of Education and Human Development. ERIC Clearinghouse on Higher Education (ERIC Document Reproduction Service No. ED378924)

Lisk, A. R. (2000). Using teams to enhance learning in management education Retrieved March 9, 2004, from the University of Dubuque website : http://cobacourses.creighton.edu/MAM/2003/papers/lisk.doc

Powell, R. R. (1992). Acquisition and use of pedagogical knowledge among career-change preservice teachers. Action in Teacher Education, 13(4), 17-24.

Saavedra, R & Kwun, S. K. (1993). Peer evaluation in self managing work groups. Journal of Applied Psychology; 78(3), 450-462.

Sapin-Piane, B. M. (1993). Mature adults becoming teachers: Sailing towards Ithaka. Unpublished doctoral dissertation, University of Massachusetts, Amherst.

meaningful. In this way, critical and supportive feedback is emphasized while evaluation is de-emphasized.

Methodology

This project involved the use of one graduate educational research class. The subjects in the course were nineteen (19) teachers who, when asked to rate their skills at conducting research, showed a range of scores from 3 to 10 with 6 being the median score and 5 being the mode score (see Table l). Most students expressed that they knew there was much more to learn about conducting research.

An educational research text by Gay and Airasian (2000) served as a guide for the instruments designed for this study. Prior to the beginning of the activity, rules for completing the peer evaluation forms were shared and discussed with the class.

Each student was given forms to share with two peers of his/her choice in the evaluation of the research paper. Each student provided a copy of the partial research paper to the two self-selected classmates to review and submit suggestions and/or comments about the paper. Each reviewer signed the review sheet to indicate who had reviewed the research paper for the researcher. In this way, the researcher could ask for clarification from the reviewer about suggested revisions prior to making final revisions to the paper.

The evaluation form was divided into sections that corresponded to the portions of the paper that were to have been completed at the midpoint of the course. Specific questions about the problem statement, review of related literature, hypothesis and descriptions of the participants were included on the review form. For example, a question about the "Review of the Related Literature" section asked the following: Is the review comprehensive? Although, the peer consultant had not reviewed the information for the topic, it was expected that the written information provided by the student researcher would be comprehensive enough for the reviewer to understand the topic. Since each student in the course had completed similar research papers each reviewer was knowledgeable about the type of information that was to be included in each section of the paper.

A column on the evaluation form was included for the peer reviewer to make comments relative to each question. After the forms were returned, the student researcher reviewed the comments and talked with each reviewer concerning comments needing clarifications. The student researcher then composed a paragraph about the lessons learned from the activity and rated the activity on a 5-point Likert scale (1=strongly disagree to 5=strongly agree) on four statements relative to the activity. Lastly, a questionnaire was developed for the peer evaluators to make comments about the activity.

This learning activity was administered during the mid-term examination week, approximately halfway through the semester. At the end of the activity, each student submitted the original research paper with peer review sheets and the student researcher form attached. Each student also submitted the questionnaire that evaluated the activity.

Findings

Meaningful Peer Feedback

To address this area, the student researchers were asked to respond to three questions:

1. Was the feedback helpful?

2. Was the feedback constructive?

3. Was the feedback clear and understandable?

All of the student researchers agreed that the feedback was helpful, constructive, and clear and understandable. They felt most strongly about the constructiveness of the feedback (84% strongly agreed) and least strongly about how clear and understandable the feedback was (only 58% strongly agreed). (See Table 2)

In addition to the above, the peer evaluators were asked if they were able to give constructive feedback and whether or not they felt that their feedback should be useful in helping the student researcher revise his/her paper. All of the peer evaluators responded positively to both questions. Although 100% of them strongly agreed that they gave constructive feedback, only 79% strongly agreed that the student researcher should be able to use their feedback to revise the research paper. (See Table 3)

Based on the responses of the student researchers and peer reviewers, the peer evaluation activity was meaningful and effective in this case. The student researchers found the feedback to be constructive and clear and therefore useful and the peer reviewers were confident that their feedback was constructive and helpful to the student researcher.

Peer Evaluator Lessons

The peer reviewers were asked if they understood their role as evaluators and 100% of them agreed they they did (68% strongly agreed). However, the responses of the student researchers who also served as peer reviewers were more illuminating. The comments can be categorized into the areas of "content" and "process". Statements about the content dealt with the research topic and the material written to support the stated hypothesis. Statements about the process dealt with the evaluation of the activity.

Example statements of "content" were as follows:

* I must work harder to cover every aspect of this paper so that the reader will not wonder about any validity of my hypothesis.

* I need to analyze and critique more studies for my paper.

* I need to define more words for the reader.

* It's strange how some things make perfect sense to you, the author, but aren't quite clear to another person reading the article.

* I'm not sure I agree with some of the suggestions from my reviewers.

* I learned a lot about other topics, like retention and the use of the Lightspan program.

Example statements of "process" were as follows:

* After I received my peer evaluations, the evaluators suggested I change the same things I was already considering changing.

* It's hard to look at your own paper objectively.

* I enjoyed this more than I thought I would. I was able to see that I actually knew what I was doing and that I was getting my points across.

* I am learning that we are all struggling together. I do need the critiques of others because if 2 people see that something is missing, I know it is not there.

* The job of critiquing is not easy.

* I was able to view the other students' papers from a position of critic that increased my level of researcher adeptness.

* From looking at and reviewing other research papers, I've learned a lot about the process and how things should flow.

It is clear from the comments that the students learned things about the papers as well as about the research process. Some students realized that clarity is extremely important but not always easy to achieve. Some students also found that the review process helped them do a better job as researchers since they gained a better understanding of how their papers would be reviewed.

Impact of the Peer Evaluation Activity

All of the students indicated that they intended to use the feedback to revise their papers and that they felt that the activity helped them become better researchers.

Overall the students in this course felt that peer evaluations were helpful, meaningful and effective. I noticed that many of the students were very cautious of the comments that they made on their peers' papers. Many asked me if it was all right to mark on the paper and to write certain comments. I felt that these comments stemmed from their own fear of receiving written comments on their papers. They were likewise very apologetic to their peers when clarifying their written comments.

Similarly, the students as a whole were still "learning the process" of conducting research and welcomed comments from someone other than the instructor. They viewed the peer evaluator as someone who knew the process as they knew it. It seemed to help to have another set of eyes to view their work. The ensuing discussion between the student researcher and peer evaluator also allowed the researcher to clarify any content that was unclear or confusing to the peer reviewer.

The peer evaluators felt that this activity was helpful in helping them in the final revisions for their papers. They felt that the activity was a good way to evaluate their understanding of the information that they had learned up to that point in the semester. Likewise, the students felt that it allowed them to be "facilitators" to others learning the process of conducting research.

In general, this peer evaluation activity proved to be very effective as another way to evaluate student research papers. It allowed the students to make corrections and revisions that were from their peers rather than the instructor. It gave them an opportunity to defend their research design and statements that they had written. It also allowed the instructor an opportunity to evaluate their skills as evaluators. The student researchers stated that the activity would make them better researchers.

Overall, the quality of papers from this class was significantly higher than papers collected from previous classes. This was particularly true of the review of literature section. The peer evaluation activity helped students realize that they has to include more substantial information in this section. The selection of information for this section was more representative of current information in the field and was organized in a more logical manner than in the papers presented before the peer evaluations were initiated.

In terms of the editing process and whether it continued to damage their self-esteem, this peer evaluation process helped to remove the stigma associated with the instructor editing their papers. The peer evaluation process allowed the students to experience the editing process by their peers as well as perform the editing of their peer's paper. In some strange way, it seemed to bring meaning to the value of "editing marks". As one student commented, "I am learning that we are all struggling together". As a researcher, I too am struggling to learn better ways of helping student researchers become better in the process of "doing research".

References

Barrett, J. (1986) The evaluation of teachers. (ERIC Digest No. 12). Washington, D.C.: ERIC Clearinghouse on Teacher Education. (ERIC Document Reproduction Service No. ED278657)

Benshoff, J. (1988). The effects of the Structured Peer Supervision Model on overall supervised counseling effectiveness ratings of advanced counselors in training. Dissertation Abstracts International, 49, no 11A. Accession No: AAG8902828

Boulmetis, J. and Dutwin, P. (2000). The ABCs of evaluation. San Francisco: Jossey-Bass

Boyd, R. T. C.(1989). Improving teacher evaluations. (ERIC Digest No. 111), American Institutes for Research Washington D.C.: ERIC Clearinghouse on Tests Measurement and Evaluation. (ERIC Document Reproduction Service No. ED315431)

Darling-Hammond, L., Wise, A. E., & Pease, S. R. (1983). Teacher evaluations in an organizational context: A review of literature. Review of Educational Research, 53(3), 285-328.

Dochy, F.J.R.C. and McDowell, L. (1997). Assessment as a tool for learning. Studies in Educational Evaluation, 23, 279-298.

Gay, L. and Airasian, P.W. (2002). Educational research: Competencies for analysis and applications. (6th edition) Upper Saddle River, NJ: Prentice Hall.

Keig, L. and Waggoner, M. D. (1993). Collaborative peer review. The role of faculty in improving college teaching. (ERIC Digest 071). George Washington Univ. Washington DC. School of Education and Human Development. ERIC Clearinghouse on Higher Education (ERIC Document Reproduction Service No. ED378924)

Lisk, A. R. (2000). Using teams to enhance learning in management education Retrieved March 9, 2004, from the University of Dubuque website : http://cobacourses.creighton.edu/MAM/2003/papers/lisk.doc

Powell, R. R. (1992). Acquisition and use of pedagogical knowledge among career-change preservice teachers. Action in Teacher Education, 13(4), 17-24.

Saavedra, R & Kwun, S. K. (1993). Peer evaluation in self managing work groups. Journal of Applied Psychology; 78(3), 450-462.

Sapin-Piane, B. M. (1993). Mature adults becoming teachers: Sailing towards Ithaka. Unpublished doctoral dissertation, University of Massachusetts, Amherst.

Nancy Reese-Durham, Ph.D., Department of Middle Grades, Secondary and Special Education, School of Education, Fayetteville State University.

Correspondence concerning this article should be addressed to Dr. Nancy Reese-Durham, 1200 Murchison Road, Fayetteville, NC 28301: Email: [e-mail].

Table 1

Self Report of Research Skill on a
Scale of 1-10, with 10 = high

Number   Research Skill

1        6.5
2        4
3        7
4        7
5        7
6        7
7        6
8        5
9        5
10       5
11       5
12       10
13       5
14       5
15       3
16       4
17       6
18       4.5
19       6.5
Median   6
Median   5

Table 2

Student Researcher Responses (N=19)

Statement                          SD   D    U    A     SA

1. This activity was helpful       -    -    -    32%   68%
  in my revisions for this paper
2. The feedback from my peers      -    -    -    16%   84%
  was constructive
3. The feedback from my peers      -    -    -    42%   58%
  was clear and under-standable

Table 3

Peer Evaluator Responses (N=19)

Statement                         SD   D    U    A     SA

1. I understood my role as a      -    -    -    32%    68%
  peer evaluator
2. I was able to give             -    -    -    -     100%
  constructive feedback
3. My feedback should be useful   -    -    -    21%    795
  for revising the paper