Evaluating Placement and Developmental Studies Programs at a Technical Institute: Using ACT's Underprepared Student Follow-up Report.

Technical schools and community colleges serve very diverse student populations, many of whom are high-risk students underprepared for college studies (Wattenbarger & McLeod, 1988; Newman, 1994; Roueche & Roueche, 1993; Smittle, 1995). Because of the populations they serve, the need for effective entry course placement is of paramount importance at these colleges (Rounds & Anderson, 1984; Hughes & Nelson, 1991; Maxwell, 1994; Newman, 1994; Smittle, 1995), especially when considering that many vocational and technical colleges have open-door policies and may be self-selected by students "for a variety of reasons including weaker secondary preparations, lower cognitive abilities, failure to complete a high school program, limited aspirations or the desire for an alternative career" (Taube & Taube, 1990, p.93).

Traditional admission tests have been found to be ineffective for use as placement instruments at technical schools and community colleges (Newman, 1994). "Standardized tests have been used for initial college course placement, but with mixed or generally unsatisfactory results, prompting the development of basic skills placement tests" (Hudson, McPhee, & Petrosko, 1993, p. 5). The American College Testing Program (ACT) began piloting the ASSET in California in the early 1980s (Rounds & Anderson, 1984). The ASSET is a student success system for two-year institutions designed to be a student advising, placement, and retention system. It includes 25-minute placement tests in writing, reading, numerical skills, elementary algebra, intermediate algebra, college algebra, and geometry. Today it is used widely by technical and community colleges for course placement (American College Testing Program, 1990).

Chattahoochee Technical Institute (CTI) in Marietta, Georgia, began using ASSET for mandatory placement in the spring of 1994. CTI administers the following four sections of the ASSET: writing, reading, numerical skills, and intermediate algebra. By the spring of 1996, student services personnel identified the need to determine whether the Institute's placement program was effective, particularly for the school's large number of high-risk students placed into developmental education classes. Thayer and Maxwell (1994) insist that colleges "must go beyond the `political correctness' of merely assuring accrediting bodies, state or federal boards and potential constituencies that they have a learning assistance program; now they must show that they have an effective program" (p. 283). Akst and Hecht (1994) agree, stating that "In an age of shrinking budgets and growing demands for accountability, the evaluation of academic programs hardly requires justification. However, the need for evaluation is particularly acute in college remedial programs" (p. 289).

CTI was the first postsecondary institution to use ACT's new Underprepared Student Follow-up Report (UPSFUR) service. The UPSFUR is a means through which institutions who use the ASSET for course placement, particularly developmental studies course placement, can assess whether students placed into the courses are successful in subsequent courses. Course placement is compared with subsequent grades in the developmental studies courses and the regular level courses that follow. The student services personnel at CTI recognized the importance of conducting such a study because large numbers of students were being placed into developmental studies courses based on their ASSET scores. Questions to be answered by conducting the study were as follows: Did students placed in developmental studies courses successfully complete the courses? Did students who successfully completed the developmental studies courses continue on successfully to complete regular diploma level courses and regular associate's degree level courses?

High-risk students are known to drop out of the Institute at higher numbers than students placed directly into regular diploma and associate's degree courses. Overall, retention has been a problem at the Institute, and Chattahoochee Tech received a five-year Title III grant focused, in part, on improving retention. The mandatory placement program and the developmental studies program are two of the newly implemented programs directed at intervening with these high-risk students to prevent dropouts. Morante (1987) contends that mandatory testing and placement and developmental education help institutions maintain high standards that will attract students and help with retaining students. Moreover, Hughes and Nelson (1991) surmise that "as the accuracy of placement is improved, it seems probable that student retention will also improve" (p. 46). The research described in this report was designed to investigate how well placement and developmental studies help to prevent high attrition by determining to what extent students are successful in developmental studies and subsequent courses.

Methodology

Student records from CTI's ASSET database constituted the sample. The 1,261 records contained every student tested using ASSET during one of the following quarters: winter 1995, spring 1995, summer 1995, and fall 1995. Each of the 1,261 students' enrollment records was reviewed by using Banner computer software (the Institute's main information management system). Each students' identification number was entered manually, and each enrollment record was viewed on an individual basis. Students' schedules were reviewed for their first and second quarters of enrollment. If they did not enroll in the quarter immediately following their testing period, their records were not included in the study (even if they may have enrolled a quarter or two later); if they enrolled before the testing period, their records were not included in the study. Records of students who never enrolled were deleted from the study.

From the remaining records, students' names and identification numbers were recorded on grid sheets provided by ACT, along with grades for selected developmental studies and regular courses. Selected courses were ENG 096 (lowest level developmental), ENG 098 (third and highest course in series of developmental), ENG 101 (diploma level regular), ENG 111 (diploma level regular business), ENG 191 (English Composition I, associate's degree level), MAT 096 (lowest level developmental), MAT 099 (elementary algebra), MAT 101 (basic math, diploma level), MAT 111 (business math), and MAT 191 (college algebra). ASSET scores were matched by identification numbers from ACT's records when the UPSFUR statistical analysis was completed. (Each quarter all ASSET scores are downloaded to a disk and sent to ACT.)

The data collected for the study had some limitations. The sample sizes for some courses were small. ACT recommends that results based on small sample sizes (under 40) be interpreted with caution. Moreover, students who were admitted based on criteria other than ASSET scores (ACT, SAT, TAPP, or transfer credit) were not included in the original sample of 1,261. It is likely that college prep students took ACT or SAT in high school. However, the majority of students who enroll in the Institute take the ASSET placement test prior to entry.

Data (on the grid sheets) were sent to the American College Testing program offices in Iowa City, Iowa, for analysis. ACT prepared a report including diagrams for each course pair (a developmental course and its subsequent regular level course). The charts indicated, among other things, percentages of students who qualified for certain courses and enrolled in the courses. The reports also displayed grade distributions by percentage. The charts included mean ASSET scores for students enrolling in each course and the probabilities for success (with a grade of C or higher) based on the ASSET score.

Results and Discussion

Tables 1 through 5 display data regarding numbers of students who were placed into and subsequently enrolled in developmental studies courses (096, 098,099) or regular level courses (101, 111, 191) based on their ASSET scores, and also include grade distributions by percentages. Developmental studies grades are satisfactory or unsatisfactory (S/U) whereas regular level courses establish grades A through F; in both cases, W represents a withdrawal from the course, either by the student or by faculty recommendation.

Table 1
Student Grade Distributions for MAT 099 and MAT 191

                                      MAT 099           MAT 191

                                     S    U    W   AB    C   DF    W

Students placed into 099 (n=30)     67   17   17   53    7   13   27
prior to 191 (n=15)

Students placed directly into 191                  62    0   31    8
(n=13)

Note. Values represent percentages of students.

Table 2
Student Grade Distributions for MAT 096 and MAT 101

                                       MAT 096           MAT 101

                                     S    U    W   AB    C   DF    W

Students placed into 096 (n=83)     61   22   17   86    7    0    7
prior to 101 (n=14)

Students placed directly into 101                  73   17    6    1
(n=108)

Note. Values represent percentages of students.

Table 3
Student Grade Distributions for ENG 098 and ENG 191

                                       ENG 098          ENG 191

                                     S    U    W   AB    C   DF    W

Students placed into 098 (n=47)     68   21   11   13   38    6   44
prior to 191 (n=16)

Students placed directly into 191                  45   18   13   24
(n=55)

Note. Values represent percentages of students.

Table 4
Student Grade Distributions for ENG 096 and ENG 101

                                         ENG 096          ENG 101

                                     S    U    W   AB    C   DF    W

Students completing 096 (n=53)      70   19   11   17   42   17   25
prior to 101 (n=12)

Students placed directly into 101                  54   21   11   14
(n=85)

Note. Values represent percentages of students.

Table 5
Student Grade Distributions for ENG 096 and ENG 111

                                        ENG 096         ENG 111

                                     S    U    W   AB    C   DF    W

Students completing 096 (n=53)      70   19   11   40   40   20    0
prior to 111 (n=5)

Students placed directly into 111                  56   22   11   11
(n=85)

Note. Values represent percentages of students.

The results indicate that students are, for the most part, successfully completing the developmental courses in which they are placed based on lower ASSET scores. The successful completion rate (61-70%) is adequate. Roueche and Roueche (1993) define successful remedial programs as those having 50% or higher retention in developmental studies courses. In addition, the results indicate that students who are required to take developmental courses prior to enrolling in regular courses are succeeding in regular courses with similar completion rates, but with lower grades than those students who were placed directly into regular level courses based on higher ASSET scores.

These findings support those of previous researchers. Miller and Rheinheimer (1996) conducted a study that focused on students who completed a developmental reading course before enrolling in regular courses. Their group comparisons indicated regular students maintained significantly higher grade point averages, but that developmental students graduated in numbers comparable to regular students. They maintained that students who complete a developmental program do make satisfactory academic progress. CTI's study confirms their conclusion that developmental students are successful in subsequent regular level courses.

   Boylan and Bonham (1994) contend that

   one measure of success of a developmental program is the extent to which it
   prepares students for success in regular college courses. If the
   developmental courses are effective, students who pass developmental
   courses should also pass regular curriculum courses in the same or related
   disciplines. (p. 307)

Their study, supported by the Exxon Education Foundation and conducted by the National Center for Developmental Education, involved 150 institutions. The findings revealed that among students who took and passed developmental mathematics with a grade of C or better, 77.2% also passed the regular college mathematics course with a grade of C or better. For those students who passed developmental English with a grade of C or better, 91.1% passed the regular English course with a grade of C or better. Another finding of Boylan and Bonham's study resembles a finding of the Chattahoochee Tech UPSFUR study. These researchers concluded that "for the most part, the grades of developmental students lagged somewhat behind the grades of other students throughout their academic careers" (p. 309). This is no surprise, they say, because these students were judged to be underprepared in the first place.

In contrast, Napoli and Hiltner's (1993) findings differed from those of the CTI study, Miller and Rheinheimer, and Boylan and Bonham. Napoli and Hiltner reviewed records of 16,000 students who enrolled in reading remediation courses. They found that these students' later grade point averages were superior to students who had not taken reading remediation courses. They wrote that their "findings suggest that the developmental reading program not only accomplishes its goal of preparing students to meeting the demands of content-area coursework but, in addition, elevates performances to a level above those of nondevelopmental students" (1993, p. 16).

Likewise, Fleischauer (1996) found that a reading and study skills course at Edinboro University had "a positive and lasting effect on the students who enroll. The students who enrolled in the courses not only had higher grade point averages than their peers who avoided it, they also suffered less attrition." (1996, p. 22). The study compared groups of students with comparably low SAT scores; one group took the developmental study course and one group did not.

White (1995) examined the data of two studies conducted in New Jersey and California regarding placement and remedial writing programs. He found that "the effect of a placement program, followed by a careful instructional program, is to allow many students who would otherwise leave school to continue successfully in the university" (p. 75). These studies did not address the issue of subsequent grades but rather retention.

This is an important point for Chattahoochee Technical Institute because one of the reasons for instituting the placement and developmental programs was to increase retention. However, the UPSFUR study does not specifically address retention rates. In some cases, the number of students who enrolled in a regular level course after successfully completing a developmental course is quite low. For example, 53 students enrolled in ENG 096; 37 successfully completed. Only 5 subsequently enrolled in ENG 111. Twelve of the 53, however, enrolled in ENG 101, and still others may have enrolled in ENG 097 (those in the associate's degree track). There are still students who did not re-enroll in the Institute. Their attrition causes concern for the Institute. A different study would be necessary to find out why those students are dropping out or stopping out.

Implications and Recommendations

Chattahoochee Technical Institute can with confidence continue its placement and developmental studies efforts with the knowledge that it is as successful as other similar programs, although there is room for improvement. Over 50% of the students at the Institute who are placed in developmental or regular level courses based on ASSET test scores satisfactorily complete the developmental studies classes and earn grades of C or higher in the regular courses. Students who complete the developmental studies programs in math and English are successful in subsequent regular level courses, albeit with lower grades than their regular student counterparts. Boylan and Bonham (1994) maintain that little information has been accumulated on the effects of developmental programs on cumulative grade point average, long-term retention, or subsequent student performance in regular college courses. ACT's Underprepared Student Follow-Up Report (UPSFUR) is a simple way for colleges who use the ASSET to begin evaluating their placement and developmental programs.

The information gained most certainly will be pertinent to some other institutions, especially those similar in nature to Chattahoochee Technical Institute, a suburban institute located 15 miles north of Atlanta. It is one of 33 technical institutes that operate under the Georgia Department of Technical and Adult Education. The Institute has an enrollment approaching 2,400 students (with the average age of students being 26) and a growing minority population (approximately 40% being non-White). With a population of students who are returning to school after an absence of 8 to 10 years, a large group of traditionally underserved students, and a growing population of students whose primary language is something other than English, effective placement and developmental studies programs are vital to the students' success. Other similar technical institutes and community colleges may want to consider participating in the Underprepared Student Follow-Up Report (UPSFUR) service or some similar type of research. It is a means of determining program effectiveness and justifying the dollars that go toward supporting costly placement and developmental education programs. It also allows institutes and colleges to identify areas where attrition problems may exist.

References

Akst, G., & Hecht, M. (1994). Program evaluation. In M. Maxwell (Ed.), From access to success (pp. 289-302). Clearwater, FL: H & H Publishing Co., Inc.

American College Testing Program. (1990). ASSET technical manual. Iowa City, IA: Author.

Boylan, H.R., & Bonham, B.S. (1994). The impact of developmental education programs. In M. Maxwell (Ed.), From access to success (pp. 121-128). Clearwater, FL: H & H Publishing Co., Inc.

Fleischauer, J.P. (1996). Assessing developmental reading courses: Do they have an impact? Research and Teaching in Developmental Education, 12(2), 17-24.

Hudson, J.B., McPhee, S.A., & Petrosko, J. (1993). The relationship between tests, course placement, and the academic performance of college freshmen. NACADA Journal 12(2), 5-14.

Hughes, R.E., & Nelson, C.H. (1991). Placement scores and placement practices: An empirical analysis. Community College Review, 19(1), 4246.

Maxwell, M. (Ed.) (1994). From access to success. Clearwater, FL: H & H Publishing Co., Inc.

Miller, E., & Rheinheimer, D. (1996). The effect of developmental reading instruction on the academic success of underprepared college freshmen. Unpublished manuscript presented at the Second National Conference on Research in Developmental Education, Charlotte, NC.

Morante, E.A. (1987). Selecting tests and placing students. In M. Maxwell (Ed.), From access to success (pp. 121-128). Clearwater, FL: H & H Publishing Co., Inc.

Napoli, A.R., & Hiltner, G.J. (1993). An evaluation of developmental reading instruction. Journal of Developmental Education, 17(1), 14-18, 20.

Newman, E. (1994). Predicting grades in basic algebra. AMATYC Review, 15(2), 47-53.

Rounds, J.C., & Anderson, D. (1984). Entrance assessment and student success. Community College Review, 12(3), 10-15.

Roueche, J.E., & Roueche, S.D. (1993). Between a rock and a hard place: The at-risk student in the open-door college. Washington, DC: Community College Press.

Smittle, P. (1995). Academic performance predictors for community college student assessment. Community College Review, 23(2), 37-45.

Taube, S.R., & Taube, P.M. (1990). Pre- and post-enrollment factors associated with achievement in technical postsecondary schools. Community/ Junior College, 14, 93-99.

Thayer, S.C., & Maxwell, M. (1994). Striving for excellence: Program evaluation through national standards. In Maxwell, M. (Ed.), From access to success (pp. 283-288). Clearwater, FL: H & H Publishing Co., Inc.

Wattenbarger, J.L., & McLeod, N. (1988). Placement in the mathematics curriculum: What are the keys? Community College Review, 16(4), 17-21.

White, E.M. (1995). The importance of placement and basic studies: Helping students succeed under the new elitism. Journal of Basic Writing, 14(2), 75-84.

Dawn Z. Hodges is an assistant professor of education and human development at Brenau University in Gainesville, Georgia ([e-mail]).