Getting Real: Implementing General Education Assessment that Works.

Abstract

Implementing assessment of general education outcomes in an institution with churning diverse enrollments and a lack of control over entering and exiting of students is a challenge. Utilization of a stratified proportionate random selection of course sections, and cooperation of teachers and students leads to a representative sample using ACT's College Assessment of Academic Proficiency (CAAP), which supports effective assessment generalizations. Closing the assessment loop by providing not only feedback to the college community but also by collecting responses to the feedback and promoting a continuing dialogue promote improved student learning and institutional change.

Introduction

Those designing institutional level assessment of general education face the challenges of balancing the ideals of "textbook perfect" research against the realities of their institution. Sometimes these conflicts result in paralysis as stakeholders fail in compromise. During the past several years faculty members on the Student Outcomes Assessment committee (SOAC) at College of DuPage (COD) designed and implemented general education assessments using ACT's CAAP (College Assessment of Academic Proficiency) combined with student tracking information in a college-wide sampling of students. The results of this assessment have been used to encourage changes ranging from individual classrooms to college-wide efforts. (Those interested in activities at classroom and discipline-program levels are encouraged to visit our web site at <http://www.cod.edu/outcomes/>.)

As a community college in suburbs west of Chicago, COD faces a college mission with a variety of expectations in serving more than thirty thousand students. One part of the COD mission involves providing general education to students with transfer (52%), graduation (33%), occupational-vocational (5%), and personal interest (2%) goals. True to the ideals of multiple indicators and measures, the College had in place several approaches to assessment of student learning and success. While consideration of persistence, transfer rates, transfer students' success, alumni and employer surveys, and degree completion serve as external indicators, faculty members sought more direct indicators of general education skills.

The challenges of general education institutional assessment can be divided into at least four sets of issues:

* One set of issues focuses on defining general education outcomes and then operationalizing that definition with appropriate instrumentation.

* A second set of issues focuses on implementing the measures defined including identifying, recruiting, and testing a representative sample.

* A third challenge is analyzing the results.

* A fourth set of issues focuses on using the results of these efforts to close-the-loop, to use the feedback to improve student learning.

Responding to these challenges COD faculty with support from administration designed and implemented four rounds of standardized testing. This effort resulted in two annual reports, which have been shared with the college community. This and other closing-the-loop activities have resulted in some observed changes among faculty and institutional responses.

Defining What's to Be Tested and How It's to Be Measured

As with any inquiry, the first steps involve identifying what one intends to study. Answering the first question, "What is general education?" is a challenge faced at any institution attempting general education assessment. Discussion and debate about the meaning of "general education" can occupy years and careers. A choice made by SOAC was to operationalize some inquiry, accepting that future discussions and alternative measures might result in other definitions. Thus, these rounds of assessment of general education utilized the following premises:

* General Education is learning which occurs as a result of multiple experiences and courses throughout a student's educational activities, not specific learning that results from a sequence of identified courses.

* General Education is defined as those ideals expressed in the following statement from the college catalog.

   The aims of general education are to enable students to understand and
   appreciate their culture and environment; to develop a system of personal
   values based on accepted ethics that lead to civic and social
   responsibility; and to attain the skills in analysis, communication,
   quantification, and synthesis necessary for further growth as a
   lifespan-learner and productive member of society.

Thus, the COD definition of general education focuses on skills, attitudes, and values; not content or knowledge. These focuses guided our implementation of assessment. While the decision to accept a current definition may not work at some other colleges, it was an important commitment that allowed us to next consider implementation.

The general education statement was broken into seven goal statements. It should be evident that there is no single procedure adequate to assess all of these outcomes. However, there is a core of general education competencies that can be identified as academic skills and which can be assessed using nationally standardized tests. There are also attitude and values questions such as "improving students understanding and appreciation of one's culture and the environment." These outcomes require self-reflective reporting not objective testing.

The assessment of skills was the first area addressed. Since general education assessment was to provide a comparison to other institutions, selection from among three national standardized tests of general skills was undertaken rather than the development of a local college specific test of competence. Members of SOAC chose CAAP based on four observations:

* CAAP provided national averages for 2-year and 4-year public colleges for both freshmen and sophomore levels.

* CAAP provided six area-tests (Reading, Mathematics, Writing skills--a multiple choice test, Essay writing--a writing sample, Critical Thinking, and Science Reasoning) that could be aggregated as valid and reliable institution level measurements. This feature contrasted with more global single tests from which subject area scores were derived. The six areas tested through CAAP addressed three (skills in analysis, communication, quantification) of the seven goal statements derived from the definition of general education.

* Each of these area tests could be administered in fifty minutes.

* CAAP was judged least dependent on specific content ideas and, therefore, was judged most likely to evaluate general learning when students were not mandated into specific core courses.

The six area-tests lined up well with a core of general education skills. The general education outcome of analysis aligns with the CAAP area-tests of Critical Thinking and Science Reasoning. The general education outcomes of communication skills can be defined as including four skills--reading, writing, listening, and speaking. Two of these skills align with three CAAP area-tests. Writing can be assessed with the Writing Skills (a multiple choice test) and Essay writing (a demonstrated essay) area-tests. The college level Reading area-test examines context reading in both the arts and social sciences. Assessment of quantification aligns with the Mathematics area-test that covers material from algebra through calculus.

Assessment of the outcomes of understanding, appreciation, and development of values in the areas of culture, environment, and ethics (the first three outcomes listed in the general education statement) is not as easily implemented with direct measures. However, the College used the CAAP option for institutional questions to begin assessment of these values and belief areas.

Three of the institutional questions on the CAAP response form were designed to begin assessment of students' perceptions in the attitudes and values areas. The form of these questions were similar to the following item:

   Based on your experiences at College of DuPage, how have your courses
   impacted your understanding and appreciation of your culture? 1. I've not
   completed enough courses to make a judgment. 2. My COD education has had no
   impact. 3. My COD education has had little impact. 4. My COD education has
   had moderate impact. 5. My COD education has had meaningful impact. 6. My
   COD education has had very significant impact.

Identifying, recruiting and testing a representative sample

At many colleges and universities one can clearly identify students as admitted to college or program, graduating or finishing a program, etc. Such is not, however, the situation with most of our students. Students enroll for a few courses, come to complete degrees, come to earn enough credit to transfer, etc. Students move on, transfer, stop-out, return, and are satisfied based on their own criteria. The college has few controls that allow for identification and even fewer that would allow us to implement entry and exit assessment.

Along with this lack of clearly identified qualification, past efforts by COD to recruit students' participation in testing have failed to produce sufficient participation despite rewards including free course work and bookstore coupons. It may be that the idea of testing is the issue since we have generated high levels of student participation in focus groups with offers of pizza and soda.

Because we concluded that mandatory college-wide testing was not feasible and voluntary testing efforts of the past resulted in high recruiting costs for low participation, a sampling model was developed utilizing testing during class time. Selected faculty are "asked" to yield one 50-minute class period to assessment. A proportionate random selection of class-sections assures testing a broad range of students. Utilizing a class session captures a high percent of targeted students with low rates of no-shows.

A high rate of participation is an important first step in developing a sample representative of the school. The testing procedure of unannounced testing during a class session results in a very high attendance rate. One check on this rate is a comparison of the number of students enrolled in the sampled course-sections and the number of test takers. Our estimate is that between 70% and 80% of the tenth day student-enrollment are tested.

But, as may be evident and perhaps already distracting to the reader, such testing will result in the assessment of a range of students including some who do not match the profile of entering freshmen and completing sophomores. Utilizing this full range of students' data is an important issue we considered when designing the analysis of these data, which will be discussed later.

Assessment tests are conducted twice a year--fall and spring. The sampling is proportionate across day-evening-weekend sections, on and off campus. In each round of assessment we attempted to gather at least one hundred tests in each subject area. Thus, over six hundred tests are administered in each testing cycle. The design calls for random sampling from all possible sections of introductory classes (100 level single classes and beginning sequences classes during the fall). During the spring the sampling was drawn from 200 level courses and end of sequences courses.

After initial identification of class-sections, the faculty assigned to each section are contacted with a request for their cooperation. Sections drawn from throughout the college included both full-time and part-time faculty. Faculty cooperation has been very high with only an occasional refusal. Testing is scheduled for a specific day and time within a two-week period. Although it is suggested that students not be told the specific date, they are informed that testing will take place. To encourage students' cooperation, they are provided a handout explaining the importance of the assessment project.

When CAAP tests are administered in the selected sections, each of the six subject-area tests is rotationally assigned to students. Thus, in any one section any student might complete any one of the six subject area-tests.

One issue of concern in considering student outcomes assessment when measured in a low stakes setting, such as the CAAP testing, is the impact of participation, motivation, and effort by students. In addition to the handout mentioned above, we have found it very helpful for the teachers to speak of the importance of this testing and to ask their students for cooperation and effort. Further we ask that the teachers remain in the classroom along with the testing associate during the testing. We've concluded that these types of symbolic supports have significant impact on recruiting student efforts.

Eighty-four percent of the respondents reported that they either tried their best or gave moderate effort, the two highest rankings on the CAAP form. This effort was consistent between the two years of testing. However, analysis of these self-reports indicates that there is also a consistent and significant difference with fall participants reporting less effort than spring participants (Fall = 81% & 83%, Spring = 86% & 86%. X2=24.5, sig. [is less than] 0.004). However, since both freshmen and sophomores may come from either fall or spring testing, this difference is not viewed as a significant bias.

Examinations of these self-reports based on subject-area tests indicated a significant dependence of effort on the subject-area test (X2=59.16, sig. [is less than] 0.000). Greatest efforts were reported for Critical Thinking (90.3%) and Writing skills (90%) followed by Math (83%) and Reading (80%). The lowest effort level for Science Reasoning was still a solid effort at 75.8%.

When considering demographic categories, on average women (86.9%) reported significantly greater effort (X2=27.26, sig. [is less than] 0.000) than men (80.1%). Age was also significantly related to effort (X2=69.03, sig. [is less than] 0.000). The general pattern indicated increased efforts with increasing age. Sixty-five percent of the participants younger than 18 reported in the first two categories compared with eighty-three percent in the 18-24 classification. In each of the six age categories from 25 to 55 and older, the self-reports ranged from eighty-seven to one hundred percent. There was no significant difference in the efforts reported by students among the race/ethnicity categories. Similarly, there was no significant difference in effort between students speaking English as their first language and those who did not speak English as their first language.

While there is some evidence of "alienation," whether focused on hostility towards schooling or testing, overall the participation and motivation of respondents was considered appropriate and acceptable.

Analysis Model--Questions Being Addressed

Comparing a cross-section of students beginning their studies with another cohort near completion of their studies is a less desirable alternative than the pre-post test model. The major limit of a cross-sectional design is that any change among subjects in skills must be more substantial in a cross-sectional design than in a pre-post test design in order to attain statistical significance. Thus, as in the case of general education, where change as subtle as one-half of the beginning standard deviation is observed, the power to detect significant change is reduced from that possible in a pre-post test model. In a non-statistical explanation, this means that the spread of scores around each cohort's average is like looking at the location of piles of salt. The average is the top of the pile and the standard deviation is the spread of the pile. In a pre-post test model, each gram of salt moves and the change in the center of the pile is observed because one is working with only the single pile that moved. In a cross-sectional model there are two piles of salt very close to each other. Each pile spreads out and overlaps with the other pile. The piles are so close and overlapping that the difference between their peaks may not be observed. This is frequently the case with assessment of value-added in general education. Unless each student is tested at the beginning and end, any change in cohort averages may not be statistically significant.

If one accepts that a clear pre-post test model cannot be implemented, but that a representative sampling of all students can be drawn using the data collection method described, then two approaches to analysis can be implemented. As with a pre-post test model, one approach uses comparisons among cohorts. We also developed a second approach involving building statistical models that explore development of general education skills. Comparison can focus on two questions:

* Are the entering and leaving students similar to national averages for similar students at other colleges and universities.'?

* Is there an indication that students have changed during their studies at an institution?

Such comparisons require identification of cases either through selection prior to testing or classification post-hoc. Given the classroom based sampling of the research design, we use a post-hoc classification procedure. Scores on the CAAP tests are matched to students' records from the tracking system. This provides us with more information than that provided from the CAAP answer form, including a record of all courses and grades. This classification approach allows definitions of eligibility to change with research questions, and it means the widest sampling from the churning student enrollments, while allowing for clean eligibility rules.

However with a general cohort sampling, it needs to be noted that such comparisons are limited in the number of cases that can be classified as appropriate to the two points used in the analysis. In these four rounds of testing, just over 600 cases out of 2,345 cases (26%) were classified as either entering freshman or completing sophomores. For the purpose of comparison, a third category was defined to include those students at a mid-point in their studies. These students might be classified as end of freshmen year or beginning of sophomore year. In this report they are labeled as mid-studies. The mid-studies classification contains an additional 26% of the cases. However, there are no national averages against which these students can be compared. This means that comparison utilizes between 26% and 52% of the collected data. If this were the end of analysis, we'd be open to justified criticism for wasteful use of our limited budget.

In order to make more efficient use of the tests collected, a broader analysis lies in constructing a statistical model that uses cases at all points along the educational development continuum. One approach to doing this lies in modeling general education skills development based on continuous development over the number and type of courses taken. Such a model uses most of the data collected in that it focuses on more than the initial and final points on the continuum of leaming. Such models, although more complex than the two-point analysis, can be very interesting. Examples of such models can be found on our web site. We are only this year beginning to explore their usefulness in the feedback loop as resources for discussion.

Using Results to Improve Student Learning

Typically, whenever results are shared at institutions, the responses range from "What are we supposed to do with this?" to criticisms based on methodology and f'mdings. In order to be effective, assessment communication needs to be a two-way process. Therefore, with each publishing of results, we included a tear-off response sheet that could also be filed electronically through our intranet. These response sheets ask multiple-choice, short answer, and open-ended questions about what the faculty members think should be done in response to the results of the assessment data. These forms are structured to ask what respondents are going to do individually as well as what disciplines, programs, and the institution can do.

For us, closing-the-loop with feedback means not only letting the college community know what outcomes assessment has found, but also asking what that community of faculty, administrators, staff, and public think about the findings, and then providing feedback on that feedback. Written responses from faculty are summarized and published (anonymously) in a follow-up document. This process allows not only for maximum input, but also maximum distribution of the reactions. After initial responses are published, there is an increase in verbal dialogue about our efforts and results.

In the fall of 1999 results, of the first cycle of assessment revealed performance significantly lower than comparable national averages in college level reading skills. It is interesting to note that after communicating these findings, reactions occurred at different levels. Some faculty reported intentions to change syllabi and course pedagogies. At that time area goals were being written for the academic year. Most of the divisions incorporated a reading-related goal in their area objectives. The findings were also a major influence in increased institutional-level support for the development of a reading center and support services at the institutional level.

It's not easy to maintain this high level of interaction. We've used four-color publications, presentations, meetings from college-wide to discipline level, the Interact, e-mail, and face-to-face contact. On occasion this has involved making a point to meet with and hear a faculty concem, then trying to address that concern. For example, a critic posited that one of the reasons for low reading scores would be a higher percent of non-English as first language students at the college. We conducted a comparison, which documented the effect of ESL students and demonstrated that they improved performance at a rate exceeding other students. This information was then shared in our quarterly newsletter.

When we published out first reports, initial response rates by faculty were low. However, as we maintained our efforts, we've experience a consistent increase in recognition and participation by faculty. These efforts demonstrate that effective assessment based in the realities of applied ideals can impact the teaching-learning dynamic.

Peter T. Klassen, College of DuPage Russell Watson, College of DuPage

Klassen, Ph.D., is Professor Emeritus of Sociology and Interdisciplinary Studies and primary researcher for the Student Outcomes Assessment Committee <p. klassen@worldnet, att. net>. Watson, Ed.D., is Professor of Psychology and faculty chair of the Student Outcomes Assessment Committee <[e-mail]>.

The authors wish to thank their dedicated colleagues on the Student Outcomes Assessment Committee, especially Janis Geesaman, Associate Dean of Liberal Arts. A version of this paper was presented at the AAHE Assessment Form in Charlotte, IV. C., in June 2000.