Using multiple raters to assess the oral communication skills of senior-level marketing and management majors.

ABSTRACT

My institution is in the process of developing an outcomes assessment plan in response to a recommendation from its accrediting agency, the Middle States Association of Colleges and Schools. The first step in the strategic plan requested that School of Business faculty develop course-embedded prototypes to test characteristics students should demonstrate in the areas of thinking and analytical skills, social skills and values, and communication skills at distinct points in their academic careers. This paper describes the use of multiple raters to assess the oral communication skills of 15 senior-level Marketing and Management students enrolled in an Advertising course. The course's final project required a comprehensive advertising proposal and provided an excellent opportunity to assess senior students' ability to polish and integrate their oral communication skills in a major field course.

1. INTRODUCTION

Numerous colleges and universities have made substantial efforts to revise and reform undergraduate curriculum and to establish student outcome assessment programs. This is often done as the result of increased national attention to improvement of student performance and stronger academic direction (Ratcliff, 1993). The advantages of assessments are that they cause us to focus clearly on what we wish to accomplish and they can be more squarely situated as part of the instructional process (Solomon, 1998). Education cannot occur apart from specific content. However, students should be able to assume that education will amount to more than just the coverage of unconnected subjects and more than exposure to information that could be acquired independent of the communities that colleges create (Challenges, 1991).

My institution is in the process of developing an outcomes assessment plan in response to a Periodic Review Report for its accrediting agency, the Middle States Association of Colleges and Schools (Framework, 1996). This issue has received a good deal of attention and most of the six regional accreditation associations have incorporated outcomes assessment as a criterion for institutional approval (Ratcliff, 1992).

Since the purpose of student outcomes assessment is to determine what and how much students learn, then an essential step in the planning process is deciding what you want students to know and be able to do upon completion of their academic program. Curriculum design has not always been given the attention it deserves in higher education and is often more influenced by the expertise of faculty than the knowledge base for which students should be responsible (Freedman, 1998). Thus, it is important that faculty become involved in and support all aspects of the assessment process since external or top-down pressures often meet resistance (Halpern, 1987). The first step in our plan was a request that faculty develop course-embedded prototypes for the testing of key components.

2. BACKGROUND

Excellence in undergraduate education is not something that just happens. It is achieved through explicit planning and, in most cases, through the intelligent employment of limited resources and the willingness to modify a planned approach to fit the actual circumstances encountered. It is important that institutions design assessment strategies to assess the ability of students to apply what they have learned to a new stimulus, rather than their ability to demonstrate information recall (Farmer, 1988).

Using the Middle States recommendation as a springboard, my institution made assessment a priority when it developed its Strategic Plan: 1998-2003. The implementation of the assessment plan had pilot projects being used to test the feasibility and success of certain key components as a first step. Assessment and evaluation activities are usually more successful if there is a shared commitment to ensure the assessment is central to and integrated within the College's regular activities and not added as a separate function performed by a new bureaucracy. Hence it is more effective to place the responsibility of assessment in individual offices with support from the Office of Institutional Planning and Research (Assessment Guide, 1998).

2.1 School of Business Outcomes Assessment Committee

The Dean of Business willingly accepted the responsibility for outcomes assessment on behalf of the School of Business. An internal Outcomes Assessment Committee was established and charged to draft a plan that met the Business School's needs. According to Banta (1988), the ultimate objective of an assessment program is to track the intellectual development of students from college entry to graduation and beyond. Following this line of thinking, the Committee focused on program assessment rather than course assessment and sought to:

* Determine broad educational outcomes desirable at distinct points within the program.

* Use an assessment device to determine the degree to which these outcomes were being achieved.

* Provide faculty with assessment results so they could explore effective strategies for improving outcomes.

The Committee also focused on the characteristics Business School students should demonstrate in the areas of thinking and analytical skills, social skills and values, and communication skills. The Committee also defined the desired outcomes within each of these areas at three distinct points in a student's career:

* The end of freshman year

* The end of business core

* The end of senior year

2.2 Request for Proposals

The School of Business Outcomes Assessment Committee put forward a request for proposals to develop and/or administer prototype course-embedded assessment instruments designed to measure the following program outcomes:

* The outcomes to be measured.

* The point in the students' program where the outcomes were to be measured.

* The course(s) where the assessment would take place.

* A detailed description of the assessment instrument.

Accepted proposals were expected to offer feedback to the Committee by providing a copy of the assessment instrument, the results of the assessment, an evaluation of the usefulness of the assessment instrument, and suggestions for the future administration of the instrument.

3. OVERVIEW OF RESEARCH PROPOSAL

This proposal was designed to assess oral communication skills at the end of senior year. The expectation was that students could make a compelling oral presentation in their major field, incorporating knowledge from the other functional areas as applicable, and integrating the appropriate media into their presentation. It was also expected that students could extemporaneously discuss various business related topics, and apply knowledge from the various functional areas to discussions and analyses of current business related issues.

Strong communication skills are vital in today's workplace (Thill & Bovee, 2002), and feedback-based changes that are the result of responses from the target audience are crucial to the overall improvement of these skills (Bienvenu & Timm, 2002). The study involved one section of MKMG-325- Advertising, a course that contained 15 senior-level Marketing and Management majors and had the key component of a final project that required students to work in groups of three or four and function as members of an advertising agency. Each group was assigned a local client and expected to create a comprehensive advertising proposal based upon a $100,000 advertising budget. The project concluded with a group presentation for the client that was evaluated by multiple raters. A question and answer period followed each presentation.

The assignment provided an excellent opportunity to assess senior students' ability to polish and integrate their oral communication skills in a major field course. The presentations also required that students incorporate knowledge from other functional areas while integrating appropriate media into their presentation. The question and answer period also allowed students the chance to extemporaneously discuss and integrate their knowledge of additional business related issues.

4. RESEARCH PROCEDURE

An assessment instrument was developed that utilized a Likert Scale 1-7 to assess degree of effectiveness. The assessment process utilized multiple raters in order to reduce measurement error and increase reliability. The multiple ratings were conducted as follows:

1. Instructor evaluated each group presentation.

2. Selected students used the instrument to evaluate a presentation (e.g. all students in Group #1 evaluated the presentation of Group #2).

3. All presentations were videotaped and students utilized the assessment instrument to evaluate their own presentation via videotape.

4. Outside professionals who regularly assessed oral communication skills were engaged to evaluate a group presentation via videotape.

5. RESULTS

An overall review of the findings shows that the numerical grades ranged from 83.7 to 96.3. If converted to letter grades, the range would be B to A. Thus, all raters in the study felt the groups demonstrated overall competency in oral communication skills at the senior level.

The professor grades and the peer grade mean differences ranged from .6 to 3.3 in the four cases. Both sets of grades were derived from "live" performances. The student self-grading was higher than the overall mean in three of the four cases, even though students were assured that their grading effort would not affect the final grade given by the professor. The grading done by the outside evaluators was lower than the overall mean in three of the four cases. The mean differences varied from 6.2 to 12.6. Both sets of these grades were derived from viewing videotape. A review of the videotapes showed variations in their quality. Outside evaluators indicated they had difficulty establishing a frame of reference when evaluating only a single presentation.

6. CONCLUSION AND IMPLICATIONS

Using multiple raters in order to reduce measurement error and increase reliability is a sound concept. However, in this experiment the raters were not all viewing presentations in the same context. Two sets of raters (Professor and Peer Group) viewed them as they were happening and also got to see all of the presentations. The other raters (Student Self-Grader and Outside Evaluator) utilized videotaping that varied in quality. The student self-graders were also able to view all other presentations and fully understood the context of their own presentation. The outside evaluators saw only one videotaped presentation.

In order to enhance this type of assessment it is recommended that the professor, the peer-graders and the outside evaluators conduct the evaluations concurrently. It is further recommended that the outside evaluators view all of the presentations in order to establish an appropriate frame of reference.

REFERENCES

Assessment Guide, Siena College, New York, 1998.

Banta, T. W., Implementing Outcomes Assessment: Promise and Perils, Jossey-Bass, Inc., San Francisco, 1988.

Bienvenu, S., and Timm, P. R., Business Communication: Developing Strategy, Developing Skills, Prentice Hall, New Jersey, 2002.

Challenges of Connecting Learning, Association of American Colleges, Washington, D.C., 1991.

Farmer, D. W., Enhancing Student Learning: Emphasizing Essential Competencies in Academic Programs, King's College, Pennsylvania, 1988.

Framework for Outcomes Assessment, Commission on Higher Education, Middle States Association of Colleges and Schools, 1998.

Freedman, K., "Culture in Curriculum," Reforming the Higher Education Curriculum, Ed. Mestenhauser, J. A., and Ellingboe, B. J., Oryx Press, Phoenix, 1998.

Halpern, D. F., Student Outcomes Assessment: What Institutions Stand to Gain, Jossey-Bass, Inc., San Francisco, 1987.

Ratcliff, J. L., Assessment and Curriculum Reform, Jossey-Bass, Inc., San Francisco, 1992.

Ratcliff, J. L., What We Can Learn from Coursework Patterns About Improving the Undergraduate Curriculum, National Center on Postsecondary Teaching, Learning, and Assessment, Pennsylvania State University, 1993.

Solomon, P. G., The Curriculum Bridge: From Standards to Actual Classroom Practice. Corwin Press, Inc., Thousand Oaks, CA, 1998.

Thill, J. V., and Bovee, C. L., Excellence in Business Communication, Prentice Hall, USA, 2002.

Frederick J. DeCasperis, Siena College

AUTHOR PROFILE

Dr. Frederick J. DeCasperis earned his Ed.D. from the University of Sarasota in 1995. Currently he is an Associate Professor at Siena College, Loudonville, New York, and Head of the Marketing and Management Department.