Building inter-rater reliability into performance assessment.

Abstract

The purpose of this paper is to examine issues ranging from strategic aspects of assessment planning to the more fundamental aspects of assessment such as measuring student evaluation processes. Assessment activities in the capstone course of a college of business were used as the focus of a study on inter-rater reliability among instructors and as a foundation for a discussion on assessment processes. The details of establishing inter-rater reliability for one instrument are provided, and conclusions are drawn concerning the value and limitations of this type of performance assessment.

Introduction

The main driver behind outcomes assessment in schools of business has been the accountability movement (Apostolou, 1999; Burke & Modarresi, 1999; Henninger, 1994; Kimmell, Marquette, & Olsen, 1998). Employers have been unhappy with the basic skills, particularly communication skills, of university graduates. Legislatures then have sought to hold universities accountable for student learning. Accreditation agencies, such as AACSB, are consequently held responsible for ensuring that business schools engage in assessment. Much effort has gone into what exactly business school administrators must do with assessment in order to get accredited (Kerby & Weber, 2000; Sinning & Dykxhoorn, 2001).

Accountability v. Assessment While accountability has provided a motivation behind assessment, it is important to distinguish the two concepts. Accountability refers to indicators used as a basis for resource allocation across divisions or for individual incentives. Assessment refers to indicators used for continuous improvement. Accountability indicators provide a means for outsiders to evaluate the quality of an institution or program, while assessment indicators provide a means for insiders to strengthen quality within the institution. Outcomes assessment has been lauded by some as a means to simultaneously provide a basis for both program improvement and accountability (McCoy & Chamberlain, 1994). More recently, however, there has been a recognition of the danger of confusing assessment processes with accountability indicators. To be effective, the tools required for each are different (Wellman, 2000). Accreditation bodies are looking for evidence that assessment data is being systematically collected and, more importantly, being widely disseminated and used effectively (Kimmell, Marquette & Olsen, 1998). Alternatives, such as seeking quantitative criteria from all universities on a series of standardized tests, for example, would probably do little to improve student learning and might instead actually impair student preparation for future careers (Henninger, 1994).

Apostolou's (1999) thorough review of the assessment literature relevant to accounting programs suggests that there is no one best assessment measure and that multiple measures of outcomes should be used to obtain feedback on programs. Most research on the validity and reliability of assessment measures has focused on general education outcomes. Very little has been published on the successful development of instruments to measure learning in business programs specifically.

Method

Faculty at a large public university in Southern California agreed to participate in this study to evaluate inter-rater reliability of a capstone course assessment instrument. Data were collected from three faculty members, two of whom were instructors in a capstone business course and one faculty member who was not an instructor of the course. One faculty-rater, a course instructor, teaches accounting, and two faculty-raters, one course instructor and one outside rater, teach primarily in management.

The capstone course emphasizes integrating business concepts from each of the business functional areas covered in the shared core required of all business administration majors and applying these concepts in a simulated business game. Students are organized into cross-functional teams formed at the beginning of each course on the basis of their declared areas of emphasis in the business administration program. Student teams develop a two-year business plan mid-way through the sixteen quarterly decisions of the simulation. The business plan was evaluated based on the quality of writing, the depth of analysis, and the effectiveness with which software and quantitative techniques were used in developing sales forecasts, production schedules, and pro forma financial statements. The faculty-raters evaluated a total of 28 projects completed by student teams from sections of the capstone business course taught in one of several terms from Fall 1999 through Spring 2000. Criteria (i.e., writing, analysis, forecasting, production schedules, and financial statements) were evaluated by faculty-raters on a 4-point scale where 4 indicated complete and excellent work and 1 indicated serious problems or missed requirements.

Data Analysis After evaluating each project, faculty-raters recorded scores on grading sheets. The grading sheets were collected and analyzed for inter-rater reliability across all three faculty-raters and for the two instructor-raters. Inter-rater reliability was calculated by counting occurrence of exact agreements of scores among raters. The number of agreements was then divided by the sum of agreements and disagreements (Bijou, Peterson, & Ault, 1968). Exact agreements were considered as those in which raters provided the same score for evaluation criteria. For example, to calculate inter-rater reliability in a scenario in which all three raters evaluated a team's writing with a score of 3, then 3 exact agreements would be counted. If, however, two raters gave the team a score of 3, but one rater scored the team's writing with a 2, then 2 exact agreements would be counted as the scores 3, 3, and 2 indicate only two raters were in agreement. An additional comparison was made by counting the number of agreements among raters when raters were within at least 1 point of the highest score. For example, if one rater evaluated a team's writing with a score of 3, but two raters gave the team a score of 2, then 3 agreements would be counted as the scores 2, 2, and 3 are within 1 point of each other and the highest score. If, however, three raters gave respective scores of 2, 3, and 4, this would count as 2 agreements as the score of 4 is the highest score, and the scores 3 and 4 are within one point of each other. The score of 2 is not within 1 point of the highest score and would not be counted as an agreement. After inter-rater reliability was calculated, the frequency of reliability scores was counted.

Results and Findings

Reviewing the data in the broadest terms, that is looking across 140 possible data points (i.e., 5 factors comprising the evaluation criteria for 28 student projects), the two course instructors scored exact agreements with each other 72 times, or 51 percent. When counting scores from all faculty-raters, there were 36 scores of exact agreement or 26 percent. Adding an outside faculty-rater decreased the frequency of 100 percent agreement considerably. In the rating scenario in which agreement was considered as any score within I point of the highest, frequency of perfect reliability scores increased. There were 129 scores of agreement within I point of the highest score between the two instructor-raters (92 percent) and 114 matches among all three faculty-raters (82 percent). Again, there was a drop in frequency of matches from 92 to 82 percent upon the addition of an outside faculty member.

Of 140 possible scores, the outside faculty-rater and instructor-rater, both with backgrounds in Management, agreed with each other 60 times or 43 percent. When counting agreements as any score within 1 point of the highest, there were 128 agreements or 91 percent. Comparing the scores of the Accounting instructor-rater with those of the outside Management faculty-rater, there was exact agreement 67 out of 140 times, or 48 percent. When considering agreement as scores within 1 point of each other, the outside Management faculty-rater and the Accounting instructor-rater agreed 129 out of 140 times, or 92 percent. Overall, the frequency of agreement is quite similar between sets of raters regardless of discipline or experience teaching the course. The two instructor-raters, one in Accounting and one in Management, had a frequency of exact agreement 51 percent of the time while the two Management raters had exact agreement 44 percent of the time. When looking at agreement of raters' scores within 1 point of each other, the two instructor-raters agreed 92 percent of the time while the two Management raters agreed 91 percent of the time.

In reviewing specific evaluation criteria such as writing, analysis, and forecasting, it was found that writing had the lowest frequency of exact agreement between the two instructor-raters, with agreement only 13 out of 28 times (i.e., 1 score for each of 5 criteria for 28 possible scores), or 46 percent. Evaluation of forecasting had the lowest frequency of perfect reliability scores for all three faculty-raters with 5 of 28 scores resulting in exact agreement or 18 percent. However, when using the less restrictive definition of agreement, that is, any score within 1 point of the highest, forecasting had the highest frequency of agreement, 100 percent. Evaluation of production schedules had the highest frequency of agreement in both rating scenarios with 16 of 28 possible scores in exact agreement between the two instructor-raters, or 57 percent, and 11 of 28 possible scores in exact agreement among all three faculty-raters, or 39 percent.

Discussion

Broad perspectives of the assessment process were delineated and a specific study of inter-rater reliability among faculty in regard to student projects was completed. In the following, we will examine questions which arise out of this research: What value was created and was this method of assessment cost-effective? What are the broader implications for developing reliable instruments for performance assessment? How do we create a culture of evidence with information feedback loops to faculty that ensures that the effort taken in data collection is not wasted? What are the implications for performance assessment and accountability?

Value In terms of costs for this embedded assessment instrument, the amount of faculty time involved in filling out the rubrics was minimal. Additional time was needed to gather the data. In future, sampling could be used to examine results from one term per year, for example. In a capstone course with approximately six to eight sections of 30 students each per term, running four terms per year, this provides a sufficiently large sample to draw conclusions about student achievement in the program on a regular basis. This instrument is thus relatively cost-effective.

Achieving Reliability Once inter-rater reliability has been measured, the next question is what level of reliability is required for Professors to have confidence in using this instrument and how much effort is required to achieve this? In this study, student projects from a team-taught course were evaluated by two instructor-raters and a third faculty-rater who had not taught the course. The frequency of reliability scores of 100 percent, indicating exact matches among all three faculty members, was quite low. Focusing on frequency of reliability scores of only two instructor-raters improved the results, but only slightly. In fact, the frequency of perfect reliability scores between the two instructor-raters was close to that of the frequency of perfect reliability between the outside faculty-rater and either instructor-rater.

Frequency of reliability scores was compared for the two course instructors and among three faculty-raters. In addition, comparisons were made for each pair of raters. Of course, adding a third rater decreased the frequency of agreement, but in comparing frequency of agreement between pairs of raters there was little difference. That is, frequency of agreement was similar between the two instructor-raters, the outside Management faculty-rater and the Management instructor-rater, and the outside Management faculty-rater and the Accounting instructor-rater. The similarity between pairs of raters can be looked at in a positive or negative light.

On the positive side, these findings may indicate a reasonable similarity among faculty in evaluating student work. Faculty can be rotated in and out of the capstone course without lowering the level of reliability currently attained. On the negative side, it appears that experience with a course, developing evaluation criteria, and background of the faculty-rater made little difference to reliability of scoring student work. Whether these findings are positive or negative has to do with the expectations faculty have for reliability. Typically the rater is the focal point of measuring student learning, but the characteristics of the criteria being evaluated also made a difference in frequency of agreement.

In this study, more well-defined criteria and explicit requirements, such as those required to develop production schedules, had a higher frequency of agreement among raters. Agreement among raters on other criteria such as forecasting and writing was much lower. The necessity and effort to improve rater reliability is heightened depending upon the nature of the work being evaluated. Careful definition of evaluation criteria and training of all faculty-raters is the first step in developing an accurate and consistent measure of student learning. Ideally, training at the beginning of a course and refresher training meetings among instructor-raters would take place to ensure that evaluation criteria is interpreted similarly among raters. Discussion of differences in perspective among faculty-raters during training sessions to ensure that background, area of expertise, or personal objectives for the course were voiced and a reasonable standard by which to measure student work was developed would improve reliability.

Culture Meaningful assessment involves continuous improvement and the development of a culture of evidence. As mentioned by Sinning and Dykxhoorn (2001), faculty involvement in assessment is extremely important if it is to be successful in the long-run. Because of accreditation pressures to have assessment processes in place, there is a tendency for deans or associate deans to take this task upon them for the sake of expediency. This can lead to a heavy reliance on standardized tests, which measure only a small fraction of what students learn in obtaining their degree. Performance assessment, which can measure a much broader array of skills, requires intensive faculty involvement. It needs to be institutionalized in academic governance processes, so that this work is done on a steady basis. As a learning organization, knowledge must be stored and effectively transmitted from one group of faculty working on assessment to the next, without gaps in time which require faculty to reinvent assessment work already done before.

Accountability The process of assessment can clearly provide incremental improvements in teaching effectiveness and curricular revision. It is not clear, however, whether it is useful or destructive to use the actual outcomes for accountability purposes. In the case of the instrument used here, a decision would have to be made concerning what is a satisfactory outcome. Should we aim for an average score of 3 on this instrument? Or, recognizing the stresses of intra-team interactions and time pressure, should a 2.5 average be sufficient? Is it worth spending extra time on the particular activities we are measuring, at the expense of other learning activities, simply to push the score on this instrument higher? By defining the standards of what is acceptable, we control the definition of success or failure. In determining standards for accountability purposes, it is important to make clear who is being held accountable. Are we holding faculty accountable for ensuring that students in their programs achieve high scores on a certain set of instruments and allocating rewards accordingly? Or are we holding students accountable for their own learning and refusing to graduate students who do not achieve certain scores on selected instruments?

The importance of achieving reliability increases the difficulty of pushing performance assessment to an accountability indicator. Apostolou (1999) has suggested that, given the increased public scrutiny upon standardized exams for their potential bias or lack of fairness, using performance assessment for accountability purposes poses a threat of litigation. Clearly, refusing to give degrees to students unable to attain certain scores on particular tests or exercises opens up a host of potential problems. Yet achieving ironclad reliability may entail rubrics that are overly narrow and that are expensive to change, thereby skewing education in inappropriate directions and stifling innovation. While it is feasible to develop reliable instruments for accountability purposes, it does not appear to be desirable or cost-effective to do so. As more comprehensive measures of student learning outcomes are developed, invariably faculty members from varied backgrounds will be requested to share in the evaluation of student work. As workload increases and more faculty become involved, there may be a tendency to forego detailed training or refresher training on rating criteria and evaluation processes. This study indicates clearly that reliability cannot be taken for granted.

References

Apostolou, B.A. (1999). Outcomes assessment. Issues in Accounting Education, 14(1), 177-198.

Bijou, S.W., Peterson, R.F., & Ault, M.H. (1968). A method to integrate descriptive and experimental field studies at the level of data and empirical concepts. Journal of Applied

Behavior Analysis, 1 (2), 175-191. Reprinted in Methodological and Conceptual Issues in Applied Behavior Analysis 1968-1988 from the Journal of Applied Behavioral Analysis. (1989). The Society for the Experimental Analysis of Behavior, (Iwata, Bailey, Fuqua, Need, Page, & Reid, Eds.), 4, 83-99.

Burke, J.C. & S. Modarresi. (1999). Performance. Change, 31(6),16-23.

Henninger, E.A. (1994). Outcomes assessment: The role of business school and program accrediting agencies. Journal of Education for Business, 69(4), 296-298.

Kerby, D. and S. Weber. (2000). Linking mission objectives to an assessment plan. Journal of Education for Business, 75(4), 202-209.

Kimmell, S.L., R.P. Marquette, & D.H. Olsen. (1998). Outcomes assessment programs: Historical perspective and state of the art. Issues in Accounting Education, 13(4), 851-868.

McCoy, J. P. & D. Chamberlain. (1994) The status and perceptions of university outcomes assessment in economics. Journal of Economic Education, 25(3), 358-365.

Sinning, K.E. & H.J. Dykxhoorn. (2001). Processes implemented for AACSB accounting accreditation and the degree of faculty involvement. Issues in Accounting Education, 16(2), 181-204.

Wellman, J. V. (2000). Accreditors have to see past 'learning outcomes'. The Chronicle of Higher Education, 9/22/00.

Laura L. Whitcomb, California State University, Los Angeles
Angela M. Young, California State University, Los Angeles

Laura Whitcomb received her Ph.D. from Indiana U, Bloomington. Her teaching and research interests include strategic management, int'l business, and cross-cultural organizational studies. Angela Young received her Ph.D. from Florida State U. Her teaching and research interests include human resource mgmt., organizational relationships, and training and development.