Student Evaluations of University Teaching.

We reviewed the literature on student course evaluations and collected many sample scales and items. On the basis of a content analysis of these scales and items and a review of the existing literature, we developed a 27-item Likert scale. The purpose of this study was to identify the characteristics of effective university teachers. In particular, this study investigated the interrelationships between items on teaching rating scales with a view to identifying those items that predict effective instruction at the university level. The data reveal five items that predicted 69% of the variance in a criterion measure of teaching effectiveness.

Research into factors affecting effective teaching has taken different approaches over the years. Initially, researchers sought to identify aspects of the teacher that might be associated with effective teaching. However, factors such as teacher personality, appearance, intelligence, and gender were unrelated to student achievement and effective teachers could not be distinguished from ineffective teachers on these characteristics (Cruickshank, 1986).

Later studies examined the association between student performance and teacher behavior. For example, Feldman (1976) reviewed studies in which students either listed characteristics of superior teachers or he identified such characteristics by finding correlations between the characteristics and global ratings. These studies demonstrated the importance of the teacher's classroom behavior. He found 19 dimensions he believed formed the basis of students' beliefs about effective teaching: value of the course, teacher's interest in the course, enthusiasm, subject matter knowledge, breadth of subject coverage, preparation and organization, presentation skills, speaking skills, sensitivity to student achievement, clarity of objectives, value of supplementary materials, classroom management, frequency and value of feedback, course difficulty, fairness, openness, encouragement and challenge, availability, and respect and friendliness. In a meta analysis, Medley (1977) demonstrated that effective teachers manage their classes differently than less effective teachers: they exercise more control; apply rules consistently; and spend more time on academic tasks. Other studies have identified such teacher behaviors as subject knowledge, organization, efficiency, self-confidence, expectation level for students, and task orientation as characterizing effective instruction.

Marsh and colleagues (Marsh, 1983, 1984, 1987; Marsh & Dunkin, 1992; Marsh & Roche, 1997), using data from several sources (factor analysis, reviews of current instruments, and interviews with teachers) support the view that teaching is multidimensional. Specifically, they identified nine dimensions of teaching: learning/value, instructor enthusiasm, group interaction, individual rapport, organization/ clarity, breadth of coverage, examinations/ grading, assignments/readings, and workload/ difficulty.

These process-product studies laid the foundation for current methods of evaluation of instruction. Teacher behaviors deemed important in effective instruction are identified and included on an evaluation instrument.

The use of paper-pencil instruments to evaluate instruction of university professors has become increasingly common (Marsh & Roche, 1993; Rushton & Murray, 1985). Seldin (1993) found that the use of student ratings as evaluation tools had increased by 57% between 1973 and 1993. Evaluation instruments can be classified into three general types or formats. One format relies solely on a series of closed-ended Likert-type items to which respondents indicate their level of agreement or disagreement with statements describing faculty instruction. Rushton and Murray indicated this was the most common method of evaluating instruction. A second type of instrument uses the narrative evaluation approach in which students respond, with their own comments and suggestions, to open-ended questions about a class and aspects of teaching quality. The third type of evaluation instrument combines the previous two, with students responding to both closed and open-ended questions.

University departments conduct evaluations of instruction for two main reasons, both related to the performance appraisal process. Teaching evaluations provide data that can be used in such administrative decisions as tenure, promotions, hiring, and salary negotiations. Because teaching evaluations provide faculty with information regarding their performance in instruction, they are also used to improve faculty teaching. It is assumed that faculty will heed the information provided in their evaluations, making appropriate changes in either or both their instructional style and the content of their lectures.

An examination of the literature on the evaluation of teaching effectiveness reveals the existence of a large variety of closed-ended rating scales. These scales vary in both the number and content of items used. This diversity of items suggests a lack of information and agreement on the factors involved in effective teaching. The diversity of items probably also reflects Marsh and Roche's (1997) view that items on such evaluation scales should reflect the multidimensionality of effective teaching. A recent series of articles in the American Psychologist dealt with some of the major validity issues pertinent to teaching evaluation instruments (d'Apollonia & Abrami, 1997; Greenwald, 1997; Greenwald & Gillmore, 1997; Marsh & Roche, 1997; McKeachie, 1997). These authors suggest at least a moderate relationship between ratings of teaching effectiveness and measures of student achievement. There is, however, disagreement on whether teaching and ratings are multidimensional (Marsh & Roche, 1997) or best described by either a global factor (Greenwald & Gillmore, 1997) or a global factor with several highly correlated lower order factors (d'Apollonia & Abrami, 1997; McKeachie, 1997). There is also disagreement on whether student evaluations are affected by such potential biases as class size and grading leniency.

For this paper, the first author reviewed the literature on course evaluations and collected many sample scales and items. Based on a content analysis of these scales and items and a review of the existing literature, the first author developed a 27-item Likert scale. This scale is provided in Table 1. Items 1-26 deal with specific components of teaching, whereas item 27 ("I learned a lot in this class") is a global measure of effective instruction. Higher scores on each item reflect higher quality. The purpose of this study was to identify the characteristics of effective university teachers. In particular, this study investigated the interrelationships between items on teaching rating scales with a view to identifying the most important components of instruction.

Table 1
Course Evaluation Form and Mean Item Scores

Rating scale:
    1            2            3            4            5
 strongly    moderately    neutral     moderately    strongly
 disagree     disagree                   agree        agree

    0
 does not
  apply
                                                        X      S.D.
1.  The assigned reading material (text, papers,       4.46    0.32
    etc.) was appropriate for the course.
2.  If used, the films used by the instructor          4.34    0.56
    were incorporated into the course.
3.  When used, in-class exercises and labs helped      4.21    0.55
    me understand the material.
4.  Grading procedures were fair.                      4.26    0.50
5.  Assessment procedures (papers, tests,              4.25    3.15
    presentations, etc.) allowed me to demonstrate
    what I had learned.
6.  The instructor gave adequate feedback on written   4.01    0.57
    work.
7.  Tests and written work were handed back in a       4.75    3.47
    reasonable amount of time.
8.  Tests, papers and/or other assessments were good   4.13    0.55
    measures of course material.
9.  The instructor came to class prepared.             4.60    0.37
10. The instructor checked to see if students were     4.16    0.52
    comprehending the material.
11. The instructor's lectures were organized.          4.41    0.56
12. The instructor's lectures were well-delivered.     4.26    0.55
13. The instructor's lectures were interesting.        4.06    0.60
14. The instructor's lectures were informative.        4.37    0.41
15. The instructor facilitated discussion/             4.19    0.51
    participation in class.
16. The instructor used humor effectively.             4.49    3.04
17. The instructor demonstrated respect for            4.26    0.41
    alternative viewpoints.
18. The instructor seemed to know the subject well.    4.63    0.35
19. The instructor used examples in an effective       4.49    0.38
    manner.
20. The instructor encouraged students to develop      4.13    0.49
    their own examples.
21. The instructor motivated me to do high quality     4.09    2.69
    work.
22. I would like to take another class from this       3.95    0.73
    instructor.
23. Overall, this instructor's teaching style is       4.27    3.00
    excellent.
24. The instructor was available during posted         4.50    0.40
    office hours.
25. The instructor was willing to help when I          4.48    0.56
    attended office hours.
26. I consider this a challenging class.               4.16    0.50
27. I learned a lot in this class.                     4.17    0.54

Method

Participants and Procedure

Course evaluations were collected over a two-year period. Evaluations were taken from 161 psychology courses offered by a Department of Psychology in the Rocky Mountain region. Full-time psychology faculty taught all of the courses used in this study. As part of the normal course evaluation process faculty administered the evaluations during the second last week of the semester. According to Department policy, faculty did not remain in the room while students completed the rating form.

Evaluations from both graduate and undergraduate (upper and lower division) courses are included in the analyses. In all, 3,632 evaluations comprise the sample. Thus, the average class size was approximately 23. Although it is not possible to give an exact number, the majority of the students who completed the evaluations were psychology majors. Further, the majority of those who completed evaluations probably completed the same instrument more than once--they may have evaluated the same instructor in different courses and or they may have completed evaluations on different instructors.

Instrument

Students completed the 27-item scale provided in Table 1.

Results

As can be seen in Table 1, all of the ratings of teaching effectiveness were positive. All but one item, number 22, had a mean score of 4 or higher indicating respondents at least moderately agreed with the statement. Students gave their highest ratings for speed of response in returning graded materials, class preparation, and subject matter knowledge.

For space reasons, we have not included the matrix containing the 351 inter-item correlations. However, all but five of these 351 correlations were positive. None of the five negative correlations were either statistically significant or greater than -.02. The majority of the correlations (160 or 45.5%) were in the moderate range, .30-.70, while only 15.5% (54) were greater than .70, and 39% (137) were less than .30.

Table 2 contains the correlations between the criterion item (number 27) and the other 26 items. An examination of this table reveals that most items had a moderate to high relationship with this global measure of teaching effectiveness. The items that correlated the highest with self-report of learning were those pertaining to lecture delivery (items 12, 13, and 14), instructor preparation (item 9), and effective use of examples (item 19).

Table 2

Correlations between Measure of Student Achievement (Criterion Item) and Scale Items

Item 1           .50    Item 2          .22
Item 5           .13    Item 6          .52
Item 9           .70    Item 10         .64
Item 13          .74    Item 14         .74
Item 17          .57    Item 18         .65
Item 21          .17    Item 22         .74
Item 25          .39    Item 26         .50

Item 3           .60    Item 4          .62
Item 7           .05    Item 8          .66
Item 11          .67    Item 12         .77
Item 15          .52    Item 16         .13
Item 19          .75    Item 20         .47
Item 23          .17    Item 24         .39

In order to determine the best predictors of teacher effectiveness we conducted a regression analysis. Using a forward selection procedure, we regressed the criterion item, number 27, onto the other 26 predictor items. Five items accounted for 69% of the variability in student perceptions of learning. Informativeness of the instructor's lectures (item 14) was the first variable selected and accounted for 61% of the variability in student perceptions of learning. The next four items entered the following order: item 8 (65%); item 9 (67%); item 13 (68%); and item 26 (69%).

Discussion

We designed this study to examine the factors that predict effective instruction at the university level. We defined effective university instruction as students' perceptions of their learning in the class. The instrument used in this study to predict effective university teaching contained items gleaned from a thorough review of the literature on teaching effectiveness. Thus, there is theoretical support for the construct validity of the instrument. In all, 27 items were included in the instrument. Student respondents rated the items on a one to five scale, where one was "strongly disagree" and five was "strongly agree."

The findings from this study are in general agreement with the existing literature. They reveal that teaching effectiveness is multifaceted: there are different components to effective instruction. Further, the data support the notion that teaching effectiveness can be predicted. The regression analysis revealed a linear combination of five items that accounted for 69% of the variance in teaching effectiveness. The items were: informative lectures, whether tests, papers and or other assignments were good measures of course material, instructor preparation, interesting lectures, and students' perception of whether the class was challenging.

The five highest-loading items are an interesting combination. Individually, their inclusion in the prediction formula is not surprising. However, the order of the items is interesting. It is comforting to know that students regard the informativeness of lectures to be of the utmost importance. That the item referring to tests and other assignments as good measures of course material was the second to enter the regression equation was somewhat surprising as this item does not deal with a professor's in-class instructional performance. Perhaps the importance attached to this item reflects the significance students attach to their grades? The next two items entering the equation (instructor preparation and interesting lectures) are not a surprise, given that they refer to in-class lecturing behaviors. It was certainly pleasing to see the challenging class item entering the equation. The presence of this item suggests that professors who challenge students in class receive high ratings for teacher effectiveness.

The criterion item, student perceptions of their learning, is an inaccurate measure of their actual learning. No data on actual learning were collected. In short, the criterion is contaminated. Our guess is that students overestimate how much they learn. Also, given the self-serving bias, students who believe their learning to have been limited are more likely to blame their instructor than their own lack of ability, effort, or poor study habits and skills. Future research might focus on examining teacher ratings in relation to a measure of actual learning. There already exists a substantial literature on the relation between grades and ratings (e.g., Greenwald & Gillmore, 1997). This literature indicates there is a positive correlation between grades and ratings. The explanation for this relationship is under debate and a topic for future research. Perhaps the use of other measures of learning, such as raw scores on tests, and their association with instructor ratings would aid this discussion.

The majority of the professor ratings were very favorable. On all but one of the items, the average rating was above 4.00. Professors received the highest average rating for handing back written work in a reasonable amount of time (item 7). Only one item, "I would like to take another class from this instructor" had a mean of less than 4.00. Interestingly, while students generally rated professors highly, they would not necessarily want to take another class from some professors again. That the evaluations were so positive, suggests a truncated range of ratings. Such a range probably adversely affected the conclusions that can be drawn from the data analysis.

An interesting future research topic for the identification of effective instruction would be to examine student ratings in conjunction with qualitative comments on teacher performance. These qualitative comments sometimes elaborate on or clarify data in the ratings. These data could also be used to examine the occurrence of a halo effect. Additionally, qualitative comments could provide further data on what students regard as important components of instruction.

References

Cruickshank, D. R., (1986). Profile of an effective teacher. Educational Horizons, Winter, 80-86.

D'Apollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52, 1198-1208.

Feldman, K. A. (1976). The superior college teacher from the student's view. Research in Higher Education, 5, 243-288.

Greenwald, A. G. (1997). Validity concerns and usefulness of student ratings of instruction. American Psychologist, 52, 1182-1186.

Greenwald, A. G. & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52, 1209-1217.

Marsh, H. W. (1983). Multidimensional ratings of teaching effectiveness by students from different academic settings and their relation to student/course/instructor characteristics. Journal of Educational Psychology, 75, 150-166.

Marsh, H. W. (1984). Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology, 76, 707-754.

Marsh, H. W. (1987). Students' evaluations of university teaching: Research findings, methodological issues, and directions for future research. Journal of Educational Research, 11, 253-388.

Marsh, H. W., & Dunkin, M. (1992). Students' evaluations of university teaching: A multidimensional perspective. In J. C. Smart (Ed.), Higher education: Handbook on theory and research (Vol. 8, pp. 143-234). New York: Agathon Press.

Marsh, H. W., & Roche, L. A. (1997). Making students' evaluations of teaching effectiveness effective. American Psychologist, 52, 1187-1197.

McKeachie, W. J. (1997). Student ratings: The validity of use. American Psychologist, 52, 1218-1225.

Medley, D. M. (1977). Teacher competence and teacher effectiveness. Washington, D.C.: American Association of Colleges for Teacher Education.

Rushton, J. P., & Murray, H. G. (1985). On the assessment of teaching effectiveness in British universities. Bulletin of the British Psychological Society, 38, 361-365.

Seldin, P. (1993). The use and abuse of student ratings of professors. The Chronicle of Higher Education, 46, A40.

Dr. Eugene P. Sheehan and Dr. Tara DuPrey, Department of Psychology, University of Northern Colorado.

Correspondence concerning this article should be addressed to Dr. Eugene P. Sheehan, Department of Psychology, McKee Hall 0014, University of Northern Colorado, Greeley, CO 80639. E-mail: [e-mail].