Student Voices: Improving the Quality of Course Evaluations.

Abstract

This article explains one school's efforts at redesigning student evaluations of courses and suggests appropriate uses of student evaluations for improvement of instruction.

Introduction

In many colleges and universities, a common assessment instrument that is used to evaluate a course is a student survey at the end of the semester. One purpose of this survey is to allow students to provide feedback that may be used to improve instruction in future courses (Brookhart, 1999; Angelo & Cross, 1993; Boyd, 1989). Since faculty often teach a different course the next semester, the information that is acquired at the end of a course may not be viewed as useful in the refinement of the next course. Additionally, if the selection of the assessment instrument is completed by the institution, the given questions may not be recognized as relevant to the instructional needs of the course instructor. Poor evaluations are also often explained by the respective instructor as being a result of unmotivated students, heavy teaching loads, or invalid rating systems (Lucus, 1999). In addition, faculty have (Coburn, 1984) argued that students lack the technical expertise to evaluate course content or instructional style. This, they explain, may result in an overemphasis on the evaluation of the teachers' popularity rather than their teaching ability. Concerns have also been raised that an overemphasis on course evaluations results in grade inflation and a reduction in amount of material that is covered in a given course (Wilson, 1998).

A great deal of research has been completed that examines the validity and reliability of student course evaluations. Cashin (1995) has reported that more than 1500 articles and books have been written that address the development, design, and appropriateness of student evaluations. Based on these resources, he determined that well-designed course evaluations can provide valid and reliable results. Other researchers have provided further support for this claim (e.g., Brookhart, 1999; Peterson & Kauchak, 1982; Coburn, 1984). Researchers (Brookhart, 1999; Scriven, 1995; Howard & Maxwell, 1980) have also found that higher grades do not necessarily result in higher course evaluations. In other words, many of the concerns that have been raised with respect to course evaluations appear to be unfounded. A well-designed student evaluation system can produce valid and reliable results.

The purpose of this article is to describe how the Mathematical and Computer (MCS) department at the Colorado School of Mines (CSM) has redesigned the student evaluation system. The changes were made with the purpose of improving the quality and usefulness of the information that is collected. Additionally, this article will describe how faculty members are encouraged to consider student feedback for instructional improvement purposes.

Previous Student Evaluation System

In the fall of 1998, a number of MCS faculty began to raise concerns about the course evaluation system. For over 20 years, CSM has administered an internally developed student evaluation instrument at the end of each course. The design of this instrument, which employs a five point Likart scale, is similar to that which is used at many schools. With the exception of three questions, the students' responses indicate the extent to which they agree or disagree with a given statement. A high rating (e.g., "4") typically indicates strong agreement with a given statement while a low rating (e.g., "0") typically indicates strong disagreement. The instrument concludes with a request that the students provide additional comments on the back of the evaluation form. The questions that comprise this instrument will be presented later in this article.

The original purpose of this instrument was to provide useful feedback to the faculty concerning how to improve a given course and to provide the head of the department with an indication of the students' perceptions of the effectiveness of the course. The only students that typically provided comments on the back of the evaluations were those that were extremely dissatisfied with the course. Their comments often yielded little information that could be used to improve the course design, e.g., "This is the worst course ever." Additionally, regardless of whether an instructor received high ratings on the constructed response portion of the evaluation, they usually only received negative comments on the back of the evaluation form. A methodology was needed that would encourage all students to share their insights in a productive manner.

Current Student Evaluation System

Several changes were made to the student evaluation instrument. The first change was that three short response questions were added to the student evaluation instrument. These questions are as follows: "What aspects of instruction did you find effective for promoting your learning?"; "What recommendations would you make that would improve the instruction that you received in this course?"; and "If you have any additional comments, please write them in the space below."

The ordering and phrasing of these questions is purposeful. Students are first asked to state the aspects of instruction that have been effective for promoting their learning. A benefit of this ordering is that when the instructors receive the students' responses, they are first confronted with positive comments. The mere existence of this question, as well as when it appears, is also beneficial. Both the course instructor and the head of the department are now provided with feedback as to what components of the course are valued by the students. Without this type of information, an instructor may unknowingly eliminate an instructional activity that is having a positive impact on student learning. The second question asks students to make recommendations for improving the course. The phrasing of this question is purposefully designed to prompt students to provide productive, rather than offensive, feedback.

Another change that has been made is that the student evaluations are now administered at the beginning rather than the end of the class period. When students are aware that they will remain in class until their instructor returns, they are more likely to use this "wait" time to complete the short response questions. In the past, the only students that were motivated to provide written comments were those that had strong, negative opinions. The new system encourages all students to participate.

Another concern of many of the faculty was that the information that was acquired through the student evaluations was acquired too late. Since the students completed the evaluations at the end of the semester, their suggestions could not be used to improve the current course. In response, the department has developed an electronic student feedback form that may be anonymously submitted at any point during the semester. Faculty members have the option of using this system in their courses.

Encouraging Faculty Members to Use Student Feedback

As was discussed earlier, faculty often explain poor student evaluations in a manner that justifies why they should not make changes in their instructional practices (Lucus, 1999). In order to encourage MCS faculty to seriously consider students' recommendations, faculty members are asked to submit a written response to the following questions: "What information did your students provide in response to the evaluations that may be useful for improving future classroom instruction?"; "What changes in instruction do you anticipate making in response to your students' feedback?"; and "What changes did you make this semester based on your student feedback from previous evaluations that you found to be effective for improving instruction?" Faculty responses to these questions are submitted and discussed with the head of the department. Both the written responses and the discussions are maintained as confidential. Responding to these questions does not guarantee that instructional change will take place; however, it does ensure that faculty members take some time to reflect on their students' comments prior to filing them in a cabinet.

Impact

The short response questions were added to the student evaluations in the spring of 1999. At this same time, instructors began submitting their responses to the students' suggestions to the head of the department. In the fall of 1999, the instructors were given the option of acquiring electronic anonymous feedback from students throughout a given semester. The original fourteen structured response questions that appeared on the student evaluations and the manner in which these questions were administered did not change over this period. This provided a common instrument in which to compare the students' evaluations of the courses before and after the described changes were implemented.

Table 1 displays the departmental average for the fall of 1998 (before the changes were implemented) and the spring of 2000 (after the changes were implemented) on each of the structured response questions in which the students were asked to indicate the extent to which they agreed with a given statement. As this table suggests, the departmental average increased for the majority of questions. In fact, there was only one question, "The instructor knows the course material," in which there was no identifiable change. A t-test was completed on each of these questions and significant changes were found on eight of these questions. Although the overall changes were small (approximately 1/10 of a point), they were all in a positive direction. In other words, the students' rated their courses more positively after the changes had been implemented.

Table 1: Departmental averages on the constructed response questions before and after the implementation of the described changes.

                                                         Spring
Question                                 Fall 1998        2000

Course material is well presented.         2.92         3.04(*)
                                         (n=1565)       (n=1537)

Complex material is well presented.        2.76         2.86(*)
                                         (n=1557)       (n=1530)

The instructor is accessible.              3.01           3.03
                                         (n=1485)       (n=1461)

Assignments are well explained.            2.82           2.83
                                         (n=1549)       (n=1521)

The instructor meets
scheduled classes regularly,               3.56         3.68(*)
except for necessary absences.           (n=1565)       (n=1542)

The instructor grades and
returns assignments and                    3.25         3.31(*)
tests within a reasonable time.          (n=1556)       (n=1540)

This instructor                            2.87         3.25(*)
facilitates your learning.               (n=1563)       (n=1530)

                                           3.08         3.17(*)
Assignments are relevant.                (n=1562)       (n=1530)

Examinations cover important               2.91         3.12(*)
rather than trivial material.            (n=1525)       (n=1516)

Grading is fair                            2.83         3.09(*)
                                         (n=1555)       (n=1536)

The instructor knows the                   3.50           3.50
course material.                         (n=1552)       (n=1525)

Note. The (*) indicates that 12 p < .05

Three of the questions on the structured response portion of the evaluation employed a different scale than did the other structured response questions. Since the validity of the student responses to two of these questions is questionable,(1) these two questions were not included in the analysis. The final question asked the following, "Overall would you consider this course instructor 4) Superior, 2) Average or 0) Poor." Although the departmental average increased from the fall of 1998 (2.90) to the spring of 2000 (2.98), a t-test indicated that this increase was not significant.

The changes that were made to the student evaluation system were in response to concerns that had been raised by members of the faculty. With this in mind, a more important outcome is the faculty's responses to the new system. Faculty members have reported that the existence of the short response questions has improved the quality of the information that they acquire from their students. The new methodology elicits feedback from students both on what is working and what is not working within the respective courses. This information allows the instructors to maintain the strengths of a course while addressing the weaknesses.

Faculty have also reported that writing a response to their students' evaluations has stimulated greater reflection on their students' suggestions. In the past, faculty often skimmed through the students' comments without considering how this information might be used. The submission of a written response has stimulated a more detailed examination of students' comments. Even when faculty members teach a different course the next semester, they can often identify general comments that can be used to improve their overall instructional approach. Additionally, faculty members who have received unusually poor evaluations within a given semester have expressed gratitude at the opportunity to explain in writing the circumstances that contributed to this occurrence. The head of the MCS department has also indicated that the students' comments have provided him with greater insight into the nature of the instruction that is occurring within the departmental courses.

Only a few of the faculty have selected to use the electronic anonymous feedback system. Over the two semesters that this option has been available, a total of four instructors have used this system in their courses. Only two of these instructors have expressed an interest in continued use of this system. The main concerns that have been expressed with respect to this instrument are the same as those that had previously been raised with respect to the original evaluation system. The only students who provided comments in electronic form appeared to be those that were extremely dissatisfied with the course and the comments that they provided tended to be structured in a non-constructive manner.

Concluding Remarks

The changes that were made to the student evaluation form and the request that faculty provide written responses to their students' feedback have been well received by the MCS faculty. The additional work that is required of faculty and students in response to these changes has been moderate. The students dedicate a few extra minutes to the completion of the evaluation forms. The faculty provide extra time in class for students to complete these forms and time outside of class to review and respond to their students' feedback. The information that is being acquired through the new system is viewed as more useful than that which was available through the old system. The students now make recommendations for change rather than disparaging remarks. Also, the new format elicits information concerning both "what works" and "what does not work." This allows the faculty to build upon the positive features of instruction while eliminating the negative features of instruction. The overall effectiveness of the new system is also supported by the observation that the students' average ratings on the structured response portion of the course evaluations have increased since the changes were implemented.

The third change in the evaluation system was not met with success. The majority of faculty have selected not to use the option of acquiring electronic anonymous feedback from their students throughout the semester. The few faculty members who have tried this system have found that it has the same problems that were witnessed in the old student evaluation system. The comments that are provided tend to be from the most dissatisfied students, and these comments tend to be stated in an unproductive manner.

The literature concerning student course evaluations emphasizes the importance of creating a positive and supportive evaluation environment (Scriven, 1995; Boyd, 1989). If faculty members perceive that their students are judging them unfairly, they will be less likely to use the information that students provide for instructional improvement purposes. MCS has found that by providing the students with guiding questions, the students can be encouraged to provide constructive feedback. This has resulted in greater faculty support of the evaluation process. Additionally, faculty members who have received poor ratings within a given semester have expressed appreciation for having the opportunity to explain any unusual circumstances that may have contributed to these results.

The evaluation system that was described in this paper can be easily adapted to other colleges and universities. The primary features of this system are the introduction of the short response questions on the student evaluation forms and having the instructors provide a written response to their students' feedback. Another important component of this system is having the students complete the evaluations at the beginning rather than the end of class. This change has resulted in an overall increase in student participation in the evaluation process. The short response questions that were added to the evaluation forms were not content specific. In other words, these same questions can be used to evaluate English, physics, chemistry, music, or many other classes. Similarly, the questions to which the instructors are asked to provide written responses may also be used for a broad range of different courses.

Endnotes

(1) One of the questions uses a three-point scale, and the students responded using a five-point scale. The other question reversed the direction of the scale. In other words, a higher score indicated less agreement. Since this did not occur on any other questions, it is unlikely that the students noticed this change.

References

Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques: A handbook for college teachers, second edition. San Francisco: Jossey-Bass.

Brookhart, S. (1999). The art and science of classroom assessment: The missing part of pedagogy. ASHE-ERIC Higher Education Report (Vol. 27, No. 1). Washington, DC: The George Washington University, Graduate School of Education and Human Development.

Boyd, R. T. C. (1989). Improving teacher evaluations. Practical Assessment, Research & Evaluation, 1 (7). Available online: http://ericae.net/pare/getvn.asp?v=1&n=7

Cashin, W. E. (1995). Student rating of teaching: The research revisited. IDEA Paper, No. 32 [ERIC Document Reproduction Service No. ED 402 338].

Coburn, L. (1984). Student evaluation of teacher performances. ERIC/TME Update Series. Available online: http://ericae.net/edo/ED289887.htm

Howard, G. & Maxwell, S. (1980). Correlation between student satisfaction and grades: A case of mistaken causation. Journal of Education Psychology, 72 (6), 810-820.

Lucus, A. (1999). Reaching the unreachable: Improving the teaching of poor teachers. The department chair. Bolton, MA: Anker.

Peterson, K. & Kauchak, D. (1982). Teacher evaluation: Perspectives, practices, and promises. Salt Lake City, UT: Utah University, Center for Educational Practices. [ERIC Document Reproductive Services No. ED 233 996].

Scriven, M. (1995). Student ratings offer useful input to teacher evaluations. Practical Assessment, Research & Evaluation, 4 (7). <http://ericae.net/Pare/getvn.asp?v=4&n=7>.

Wilson, R. (1998). New research casts doubt on value of student evaluations of professors. The Chronicle of Higher Education, 44 (9), A12-A14.

Barbara M. Moskal, Colorado School of Mines

Moskal is an Assistant Professor in the Mathematics and Computer Sciences Department and the Associate Director of the Center for Engineering Education <[e-mail]>.