Accountability and Integration in Assessment: Identifying Learning Goals.

Abstract

This paper explores a process for identifying learning goals in assessment of student learning. By actively engaging key stakeholders such as employers, institutions can identify learning outcomes important to those employers and thus address the issue of accountability. Many such outcomes may also help address the issue of what general education should include, and may thus contribute to integration of the curriculum across majors and disciplines.

Introduction

An important and timely trend in higher education is assessment. The purpose, implicitly, of assessment is not merely to generate data but to enable an institution to determine its performance and progress, and determine ways to continually improve the quality of its programs and services. Thus a variety of questions may be asked, and possibly answered, through systematic assessment. For example, an institution may wish to assess the effectiveness of its student retention efforts, or the effectiveness of its student services functions. Very commonly, assessment efforts are directed at student learning. Indeed, as Pascarella and Terenzini (1991) note, the enhancement of student learning is central to the mission of colleges and universities. Several important questions may flow from this. Are students learning what we want them to learn? What do we do if we are not satisfied with the quality of student learning? While the jargon varies somewhat from one expert to the next, the basic concept of such assessment is very straightforward: What are students supposed to be learning? How do we know if they are, indeed, learning those things? If we conclude that our students are not effectively learning what we have decided they should learn, we may need to examine possible solutions, such as curricular changes or strategies to improve teaching.

Clearly identifying what students are expected to learn and tracking progress towards those goals can be beneficial to programs and institutions in a variety of ways. One important concern that can be addressed is accountability. Programs and institutions may be accountable to a variety of stakeholders, including accrediting agencies, budgetary decision-makers, taxpayers, prospective students and their parents, and certainly current students and their parents.

In the context of accreditation, there are literally hundreds of different accrediting agencies. Some focus on very specific programs, such as business, computer science, or early childhood education. Others focus on institution-wide accreditation, such as the North Central Association. Many if not most accrediting agencies mention assessment of appropriate student learning outcomes as one of their requirements. For example, the Association for International Management (AIM) lists a number of area-specific outcomes, including those related to finance, economics, or behavioral science as well as a general education component as necessary components for business programs. AACSB also mentions systematic monitoring of each degree program to assess its effectiveness. Likewise, the North Central Association mentions "assessment of appropriate student academic achievement in all its programs and "structured assessment processes that . . . provide meaningful and useful information (NCA Commission, p. 14).

In many institutions, particularly public institutions, which are largely dependent on state budgets, clear articulation and assessment of student learning outcomes can positively impact budgetary decisions. A program or institution that can clearly state, "This is what our students are able to do upon graduation," and "This is how we track our students' progress" is better able to demonstrate the continuous improvement that may often drive resource allocation decisions. Likewise, articulating and continuously assessing learning outcomes can assist with attraction of students. If a prospective student is trying to decide between two different institutions, he or she may be more likely to enroll in an institution at which the goals are clear, and the institution's progress towards goal attainment is made public. This may also be beneficial to students who are nearing graduation. It is easier for a student to market himself/herself if he/she can say "this is what I know how to do, and here is evidence that I can actually do it."

Assessment of learning goals can also be important in the context of content or curriculum integration, another issue of interest in higher education. Boyer (1992), for example, eloquently points out that "we need.., the scholarship of integration.., we need creative people who go beyond the isolated facts, who make connections across disciplines (p. 88). In this sense, relying solely on accrediting agencies to tell us what we should be teaching or what students should be learning can be woefully inadequate. Dill (2000), for example, notes that accreditation may be too formulaic a process, and that institutional and especially professional accreditation may bypass the undergraduate core, particularly in the arts and sciences. Indeed, given that most universities purport to have some general education component, it seems reasonable that general education should indeed constitute a set of clearly defined skills or abilities, which are understood and valued by students, faculty, and other key stakeholders. This is helpful for students in understanding the purpose and value of their general education. Finally, on a very practical level, curriculum integration may be important in resource allocation: creating and offering new programs, for example, may be more feasible if there is some logical relationship amongst existing courses and programs.

Certainly, systematic assessment of student learning outcomes, if approached correctly, can be of great value in continuously improving the quality of higher education. One shortcoming of many assessment models, however, concerns a lack of attention to what students are supposed to be learning. A great deal of attention has been devoted to how we might assess students, including embedded questions, standardized testing, or portfolios. Considerably less attention has been devoted to determining what should be assessed and, indeed, some programs may rely solely on accreditation standards. Others may assess certain learning outcomes merely because there are mechanisms available to do so, not because those outcomes have been systematically determined to be relevant for that program or institution. If we are unclear about what it is we want students to be learning in the first place, how we assess their learning is of little or no value. We may generate a great deal of data, but those data may be meaningless in telling us how we are performing as an institution of higher learning. As Muchinsky (1990) notes, we cannot plunge headfirst into assessing whether students are successful until we have a fairly good idea of what we are looking for. The purpose of this paper is to describe a process for identifying some of those "whats" in a way that provides structure and meaning to the assessment process and also addresses the issues of accountability and integration.

Method and Results

The basic strategy was to focus on one group of key stakeholders in this case, area employers, to determine if there is some consistency in skills/abilities that those employers consider important. (The institution in question is a 5000-student public university located in Southeastern Wisconsin.) The rationale was that if a majority of area employers consider specific skills or abilities important, that information could be very helpful in determining curriculum content, particularly with the general education component of the curriculum. Interacting with employers was considered especially appropriate for this institution, as part of its mission specifically refers to providing educational programs that are responsive to the external communities.

The first step was to collect a sample of performance appraisal forms from area employers. (Such forms are used by a great many employers to conduct performance reviews for their employees.) Again, the premise was that if employers consider certain skills or abilities to be important enough to systematically evaluate in their employees, inclusion of those skills or abilities in the university's curriculum requirements would be beneficial for both students and for employers. Students can more easily market themselves, and employers can more easily recruit well-qualified employees.

Initial efforts resulted in a set of thirty forms and represented employers from several different Standard Industrial Classification (SIC) areas, including manufacturing (14), finance/insurance/real estate (8), retail trade (4), transportation/public utilities (1) and nonclassifiable (3). The forms ranged considerably in detail and specificity, with some employers having detailed, company- and even job-specific versions and others using very generic versions.

This initial set of forms was then reviewed independently by both authors. The content analysis resulted in identification of several broad categories of skills or abilities, including communication (mentioned in 29 forms), critical/analytical thinking (22 forms), interpersonal skills (19 forms), information literacy (6 forms), and integrity/ethics (6 forms).

Several points with respect to these categories may be important. First, the forms varied widely, as noted above, in terms of specificity and detail. With communication, for example, some employers use a form that merely lists communication as a skill to be evaluated, while others break communication down into oral, written, listening, nonverbal, and presentation. Likewise, some mention only "critical thinking" or "problem-solving" while others are much more detailed, including .judgment, decision-making, planning, prioritizing, organizing, or creativity. Interpersonal skills generally reflected such areas as managing or supervising, leadership, resolving conflicts, understanding diversity, teamwork, training/coaching, and delegation of responsibilities. Information literacy included computer skills, mathematics skills, use of data/facts, and use of appropriate reference information. In some cases, further discussion with specific employers may be helpful in clarifying what aspects of a particular skill are expected. "Basic computer skills," for example, might be further explored to determine what kinds of software or projects are desired.

Many of these skills would commonly be classified as behavioral or even as "soft" skills. Interestingly, however, some experts have argued that there is, in fact, too little emphasis on behavioral/interpersonal skills and too much on more technical skills. Mandt (1982) discusses this dilemma in the context of business education. Business programs, as noted above, often emphasize very specific content areas such as accounting, finance, and economics. Even the ostensibly more "behavioral" areas are often covered only in specific courses, such as management skills or organizational behavior. Rarely is there any integration of such interpersonal or behavioral skills into a cohesive system. Indeed, Mandt contends that such vocational education is doomed to failure: increasingly, businesses may find that they can teach technical skills themselves. Instead of focusing on such technical skills, he argues that universities might be better off emphasizing interpersonal and behavioral skills, such as working in teams, resolving conflict, and communicating effectively. The results of this pilot study clearly support his reasoning.

This same theme is echoed by Pascarella and Terenzini (1991). They note that a sufficient background of facts, data, and information content knowledge is necessary for competent and creative intellectual performance. However, they point out that merely learning factual material is inadequate; evidence strongly suggests that there also needs to be some fostering of general intellectual competence, often referred to as reasoning or critical thinking. Indeed, in their review of several decades of research on the impact of college, it is clear that content specific learning, such as that acquired in the major, is most important for the first job after graduation. After that, content knowledge becomes less and less important, and general intellectual skills and the ability to learn on the job become more important.

Second, there may implicitly be some degree of overlap in many of these categories. Effective critical/analytical thinking, for example, is of little value if the results cannot be communicated effectively. With respect to curriculum design and/or modification (as well as curriculum monitoring), therefore, it is very likely that some courses or projects may often address multiple skills. A course project may involve a group of students working as a team to conduct research on a controversial issue and write and present a report on their findings. This type of project includes aspects of virtually all of the above categories. Note that this interrelationship of skills or abilities should also be considered in assessment of student leaming. Often it is neither feasible nor desirable to attempt to assess certain skills independently of each other.

Directions for Future Research and Caveats

As noted at the outset, assessment of student leaming outcomes can provide valuable information to students, employers, budgetary decision-makers, and other key stakeholders. The most critical aspect of assessment is identifying the what. The process described here is one way to identify some of the specific skills or abilities that may be appropriate for a program or an institution. Indeed, given the emphasis on communication, critical/analytical thinking, and interpersonal skills even in this pilot study, the results may be especially helpful in addressing the issue of what general education should cover. An institution is, therefore, in a position to focus on some consistent indices of quality for all students. Likewise, students may be better able to see the value of the general education part of their education. As noted above, this type of ongoing dialogue may be especially valuable in institutions whose missions include developing and maintaining ties with the external community.

The next step at this institution will be to first broaden the scope of employers included. A more complete set of employers, representing other SIC classifications, will provide an even more comprehensive sense of some of the skills or abilities that might be relevant. A variety of factors can help determine whether a particular employer should be included in such a project. The program or institution might choose to approach employers that are currently on their advisory board, or employers that have traditionally hired their graduates. Altematively, employers who have not had a relationship with the program or institution but whose reputation or influence in the school's area is significant may also be good choices. Again, follow-up communication with some employers may be beneficial, particularly when information provided via performance appraisal forms is vague or unclear.

One important practical caveat is that this process is not intended to provide information that will stand alone. A variety of factors may influence or direct what students should be learning. Accreditation agencies in some cases may require very specific courses, topics, or experiences; those directives would then form at least part of the what of student learning. Likewise, a university may be required to follow specific mandates, perhaps from a state board of regents that dictate at least part of what will or should be learned. From a practical perspective, this process can logically address some of the what, but should not be considered as all-inclusive.

A second practical caveat concerns the potentially wide variability in the detail, and indeed the quality, of information that is available through this process. Some employers have professionally-developed, well thought-out mechanisms for evaluating the performance of their employees. Others, though, may be using mechanisms that are poor in quality and of little use even to those employers. Obviously, ongoing communication with all employers is critical to ensure that this process ultimately provides useful information.

A third caveat is somewhat more philosophical in nature. Certainly, most institutions of higher learning strive to produce graduates who are competent and have the ability to be successful. In that sense, ongoing dialogue with key employers can provide important information for curriculum content. However, universities also exist to create knowledge, not merely to reflect existing knowledge. Certainly, it is beneficial for universities to listen to key stakeholders such as employers to try and address what should be taught. It is also important for employers to listen to universities, since historically universities may advance the knowledge base in a way that will greatly benefit employers. The process described here will help ensure that there is continuous and useful dialogue.

References

AACSB. (2000). Achieving quality & continuous improvement through self-evaluation andpeer review. St. Louis, Missouri: AACSB.

Boyer, E. L. (1992). Scholarship reconsidered: Priorities of the professoriate. Issues in Accounting Education 7 (1), 87-91.

Dill, D. D. (2000). Is there an academic audit in your future? Change, July/August, 35-41.

Mandt, E. J. (1982). The failure of business education and what to do about it. Management Review, 71 (8), 47-52.

Muchinsky, P. M. (1990). Psychology applied to work. Belmont, California: Brooks/Cole Publishing Company.

North Central Association Commission on Institutions of Higher Education. (1999). Accreditation of higher education.' An overview. Chicago, Illinois: North Central Association.

Pascarella, E. T. & Terenzini, P. T. (1991). How college affects students. San Francisco, California: Jossey-Bass Publishers.

Sue M. Norton, University of Wisconsin Arthur L. Dudycha, University of Wisconsin

Norton, Ph.D., is Associate Professor of Human Resource Management and Director of Academic Assessment, School of Business & Technology <[e-mail]>. Dudycha, Ph.D., is Professor of Quantitative Methods and Vice-Chair of Business in the Department of Business, School of Business & Technology <[e-mail]>.