Using segmentation modeling to predict graduation at a two-year technical college.

This article reports the results of a longitudinal study of graduation rates at a two-year technical college where students enroll in an Associate of Applied Science terminal-degree program. The predictive ability of segmentation modeling in this study is as effective as logistic regression. However, its ability to identify discrete subgroups based on multiple independent variables is a distinct advantage over regression techniques. In addition, nonlinear relationships are easily apparent.

Introduction

The rate at which college students are retained and graduate has been the focus of considerable attention and discussion in both the academic and public arenas (Reisberg, 1999). Of particular concern is the steady drop in graduation rates at both two-year and four-year institutions (ACT, 2001).

Retention and graduation rates are a prominent topic at The Ohio State University, Wooster Campus, Agricultural Technical Institute (OSU-ATI). OSU-ATI is a small, residential, two-year technical college that operates as a branch campus of OSU. Typical enrollment is about 900 students, and most are majoring in horticulture or agriculture. The campus has an open-admission policy, and many of the students are underprepared or otherwise classified as at risk. The population is composed predominately of first-time, traditional-age, degree-seeking students who enroll full time. Most of the students live on or near the campus. The technical college attracts students who are primarily interested in, and motivated by, coursework and learning activities that emphasize applications that are career related.

Developing and offering activities and services that help improve retention is part of the staff responsibilities in both the academic affairs and student services offices. Ad hoc committees are formed periodically to address this issue and make recommendations for improving retention and graduation rates. The most recent committee charged with this responsibility developed and administered a survey to students and faculty and made several recommendations as a result (Baur & Kreps, 1996). However, Kalsbeek (1987) cautioned that such "one-time snapshots" concerning retention may reflect more of an "event perspective" than a "process perspective" and may be only "minimally useful" (p. 33).

Tinto (1993) and Noel, Levitz, and Saluri (1985) have written comprehensive texts on the subject of retention. Hyers and Joslin (1998), Mouw and Khanna (1993), and Tinto (1993) have conducted recent reviews of the literature regarding prediction of academic success in college. Schurr, Ruble, Palomba, Pickerill, and Moore (1997) also conducted a recent review of the literature regarding attrition with an emphasis on studies involving personality type.

Both academic variables, such as high school performance and standardized test scores, and noncognitive variables, such as motivation and institutional involvement, have been identified by researchers as important predictors of, and factors affecting, student success in college. Tinto (1975) developed a model which implies that students' academic and social characteristics and backgrounds strongly affect their integration and involvement in the formal and informal college environment and, therefore, their persistence.

It is well documented that, in general, students of higher academic ability and better academic preparation are more likely to persist and graduate than are students of lower ability and less preparation (Wilkie & Redondo, 1996; Tinto, 1993). High school rank, GPA, and SAT or ACT scores are traditionally used by colleges nationwide to predict the academic performance and success of matriculating students. However, both Mouw and Khanna (1993) and Tinto (1993) concluded that the ability of any academic or noncognitive variable to predict college success is very low.

One of the authors (Zimmerman) has taught (for eight years) one section of a one-credit, five-week orientation course that is required for all full-time students enrolled in Associate of Applied Science (A.A.S.) terminal degree programs at OSU-ATI. Although OSU-ATI also offers an Associate of Sciences (A.S.) transfer program, students in that program are required to enroll in a different orientation course. Students are randomly assigned to course sections with the exception of nontraditional and transfer students who are enrolled in specifically designated sections. As a course instructor, this author has access to the academic records of the matriculating students who are assigned to his sections. He also has access to the academic records of these students throughout their enrollment in the college.

The study described in this article was designed to use data on this group of students as a sample for a longitudinal study of retention and graduation at OSU-ATI. The objective of the study was to obtain quantitative data regarding the academic characteristics of the students and their rates of retention and graduation. The results of this research contribute both to the literature concerning retention and graduation and to ongoing efforts at OSU-ATI to improve retention and graduation rates.

Purpose and Hypothesis

For admission purposes, knowing which applicants have the best chances of graduating could be highly beneficial. In addition, being able to identify first-quarter students who are in good standing but who have a relatively higher risk of withdrawal could lead to intervention strategies to mitigate that risk and improve retention and graduation rates. A combination of available precollege and early college achievement variables might help clarify which students are prone to quitting and which are more likely to persist. One major purpose of this research is to assess whether a relatively new statistical technique (segmentation modeling) can simply and accurately identify graduation probabilities of both applicants and students who have completed their first quarter. Another important purpose is to explore the usefulness of segmentation modeling in identifying students who, based on orientation course grade and other academic achievement in the first quarter, are probably at risk in terms of graduation and who then could be counseled and otherwise encouraged to persist. Our research hypothesis is that segmentation modeling is as good or better than traditional statistical methods, especially logistic regression, in predicting retention and overall rates of graduation and is superior to these methods if the goal is to create groups for further analysis or treatment (i.e., counseling or intervention).

Other statistical methods exist but were deemed inappropriate for this study. Discriminant analysis is another way of predicting group membership, but the data's nonnormal distributions (see below) and nonhomogeneous variances precluded its use without necessary data transformations. Also, loglinear models could be developed but the continuous variables would first have to be converted into ordinal ones, which would lead to some loss of information. In the interest of keeping the classification scheme as simple as possible, data transformations were not attempted for this study because admissions and intervention (counseling) decisions should be based on straightforward and easily understood data.

Variables and Data

The study was conducted initially using the sample of 174 A.A.S. students who were randomly assigned by the scheduling computer to the 8 classes (cohorts of about 20-25 students) of the orientation course taught from 1992-1999 by one of the authors. These students represent a sample of about 12% of the total number of new first-quarter, full-time, traditional-age, A.A.S. students who were enrolled in the required orientation course at OSU-ATI during this period. The following four achievement variables were examined in the study: high school rank (a percentile), ACT composite score (almost all entering students take the ACT instead of the SAT), earned grade in the required orientation course (12 grade categories, with plus and minus, from F to A, inclusive), and first-quarter GPA. The two outcome variables were first-to-second-year retention and graduation rate in three years.

The achievement variables should provide a context for understanding retention and graduation. Two (high school rank and ACT score) have a precollege context and two (first-quarter GPA and orientation grade) are early college academic indicators. High school rank is an important precollege variable that may indicate ability or motivation or both. The ACT score provides an indication of a student's academic potential although it does not necessarily guarantee it. Standardized test scores, such as the ACT and SAT, are utilized extensively for college admission. The orientation course grade could be a very early gauge of the probability of retention and graduation because it is available at the midpoint of the students' first quarter, an important period in their transition to college. The first quarter GPA should be an even better early indicator of college success in terms of retention and graduation.

The types of variables evaluated in this study were both continuous (high school rank, ACT, and GPA) and categorical (orientation grade was at the ordinal scale, while retention and graduation were both nominal). The continuous variables were skewed or otherwise nonnormal according to the Kolmogorov-Smirnov test for normality (with a Lilliefors significance correction). Therefore, group differences were analyzed mainly with nonparametric procedures (Mann-Whitney and Kruskal-Wallis tests, depending on the number of groups). Differences in retention and graduation were based on the chi-square test of independence in contingency tables. Statistical results were considered significant in this study if p values were .05 or less.

The completed analysis excluded some variables which were originally considered. For example, the study excluded gender because only 22% of the sample was female (although they graduated at better rates than males: 52% vs. 35%). Also, persistence to the second quarter and graduation rate in two years were dropped because the former was very high (93%) and the latter was extremely low (7%); these variables thus provided no useful information, with the exception that most students persisted to their second quarter and so few of the sample, in fact, graduated within the desired two years. Also, dividing the students according to whether remediation was required provided no information that was not already available from high school GPA and ACT scores. Finally, since the patterns of first-to-second-year retention resembled those of graduation by the third year, a discussion of retention was excluded from the results.

Summary data for the variables and observations used in this study are displayed in Table 1 (see Appendix I). Seven cohorts (1992-1998 classes, consisting of 160 students total) formed the basis for the analysis of graduation rates because not enough time has elapsed since students in the 1999 cohort matriculated. Differences among the mean values for the individual cohorts were not found to be significant for any of the variables nor were there any significant trends over time. These mean values provide a basis for appreciating the variation among ability subgroups (see below). High school percentile rank averaged 50.2 and ranged from 0 to 98. ACT scores ranged from 11 to 29 and averaged 17.2. The mean first quarter GPA was 2.45 and the median orientation course grade was 3.0 (this variable was categorical). Only 38.8% of the population graduated in three years or less; however, of the 71.2% of the students who returned for their second year, 53.5% graduated.

Because the emphasis here is on the ability of specific variables that are known or measurable very early in the students' college experience to predict graduation by the third year, the sample was restricted to include only students who had nonmissing values of ACT, high school rank, first-quarter GPA, and orientation grade. The resulting study sample included only 139 students (instead of 160) because 21 students did not have either a high school rank or an ACT score. Table 1 shows that the means for the achievement variables did not change much by eliminating these students, and the overall graduation rate was also about the same (41.0% for the smaller sample, as opposed to 38.8%).

Statistical Procedures

Predictions of graduation were made using logistic regression and segmentation modeling. The context for these predictions is overall group or subgroup characteristics, not individual students. Both techniques, however, can help make objective decisions about admissions and intervention (for the purpose of encouraging persistence). A comparison of these procedures for univariate and multivariate cases is shown in Tables 3-5.

Logistic Regression

Logistic regression differs from ordinary least squares regression in that the dependent variable is discrete and ordinarily dichotomous. It is a standard procedure (e.g., Hosmer & Lemeshow, 1989) that is preferable to two-group discriminant analysis when the assumption of multivariate normality is not met (Press & Wilson, 1978), as is the case of this study. Logistic regression coefficients can be used to estimate the odds ratio for each independent variable in the model, or the log odds ratio of the outcome associated with a unit increase of the independent variable holding all others fixed. However, nonlinear relationships are not readily apparent with logistic regression.

Segmentation Modeling

Segmentation modeling, commonly used in marketing studies, places observations into statistically mutually exclusive subsets based on the values of independent variables. It looks for the best combinations of variables and for the best cutpoints; it will create subgroups in continuous data and combine nominal or ordinal categories if they are not significantly different. It is, therefore, useful with categorical data (ordinal or nominal) as well as continuous data and greatly facilitates the interpretation of large data sets that contain many variables. Small data sets with few variables or observations, or both, however, are not precluded for any logical reason and benefit as well from an analysis that seeks to identify patterns by subdividing a sample based on values of independent variables. Moreover, segmentation modeling can be used as a reconnaissance procedure to make possible a more efficient, in-depth analysis; it can point to follow-up procedures and indicate others that could be avoided. Segmentation modeling produces classification systems displayed in decision trees. In general, however, this type of multivariate analysis, although useful for identifying patterns of interest, was limited in this study by a small sample size that lessened statistical significance and power. This study uses the Answer Tree (version 2.1) program developed by SPSS (1998).

Because the working sample included only 139 students, the segmentation analysis necessarily produced small groups. In general, tree depth (number of levels of subdivision) was usually limited to three of four, because with more subdivisions, some subgroups (i.e., segments) would have only a few students. This study used three tree-growing methods, including Exhaustive CHIAD, C&RT, and QUEST (SPSS, 1998) to automatically generate subgroups of students based on graduation rates. These methods do not yield identical tree diagrams because they work differently and each has its own best use. More than one method can be tried easily and quickly in a search for an optimal segmentation with regard to various aspects of the research problem; no special manipulations of the data are necessary. The CHIAD method (Chi-squared Automatic Interaction Detector) evaluates all possible values of a potential predictor and merges those that are statistically homogeneous with respect to the dependent variable (Biggs, de Ville, & Suen, 1991). The C&RT method (Classification and Regression Trees) is a binary tree-growing algorithm that partitions the data into two subsets so that the observations within each are more homogeneous than the previous (parent) subset (Breiman, Freidman, Olshen, & Stone, 1984); most of the trees generated during this study, including Figures 1-3, used the C&RT method. The QUEST method (Quick, Unbiased, Efficient Statistical Tree) is another binary algorithm that was created for computational efficiency; it avoids other methods' biases that favor predictors with many categories (Loh & Shih, 1997), but it requires a nominal dependent variable.

Although the SPSS program always selects the best predictor at each step on the basis of statistical significance or amount of improvement, only small differences may rank predictors in some cases. A benefit of the program is that any significant predictor can be selected manually at any level, along any branch, or at any segment in the decision tree. Manual selection, therefore, could lead to different decision trees with about the same general significance. This kind of exploratory work is fast, very easy, and helps uncover relationships that might ordinarily be missed or which may be more relevant to a given research problem. Although manual selection is possible with logistic regression, it has to be done while initially specifying the model, not at various steps during an ongoing analysis. Logistic regression, however, could be done easily on the subgroups produced by segmentation modeling after appropriately partitioning the data file or creating specific filters.

The results reported here are for the tree-growing methods which were able to subdivide the sample and yield the best overall prediction rate for specific combinations of independent variables. Segmentation modeling is easy to do, and its visual output makes relationships obvious (Figures 1-3). Hyers & Joslin (2001) show that it works well at identifying, in a study spanning seven years, patterns of persistence at their school (a small liberal arts college in Massachusetts); it also works well for grades on student journals (Hyers, 2001) and examinations (Hyers & Anderson, 2001).

Space permits the display of only relatively simple decision trees. SPSS output for trees with more than about two or three levels and two or three independent variables can be too large for publication as part of a journal article. The text and accompanying tables, however, do summarize some of the more complex decision trees generated by this study.

Results and Discussion

Single Predictor Variables

Regarding precollege predictors, the relationship between high school achievement and graduation rate can easily be seen when the study sample is divided arbitrarily into three groups. The resulting subgroups differed substantially with regard to graduation, illustrating a positive correlation (Table 2). The graduation rate for the upper subgroup (rank > 66) was 130% and 53% more than that of the lower (rank < 34) and middle (rank of 35-65) subgroups, respectively. In logistic regression, high school rank correctly predicted 65.5% of the students who graduated by the third year (compared to 52.1% by chance and the observed 41.0%) (Table 3) but accounted for only 8.2% of the variation (as shown by the Nagelkerke [R.sup.2] in Table 3). In segmentation modeling (Figure 1), high school rank predicted correctly for 69.1% of the students (compared to 55.9% by chance) (Table 3). Sixty-one percent graduated in the subgroup with rank over 60 (33% of the sample), but the relationship of graduation and high school rank was nonlinear. Figure 1 shows that a higher rate (79.2%) characterized the subgroup with ranks from 61-76 but that the rate was less (40.9%) when ranks exceeded 76. Only 31.2% graduated in the subgroup with ranks of 60 or less (67% of the sample), and only 11.8% graduated if high school rank was 25 or less. Although trial and error with contingency tables for a single predictor would eventually yield the same results as this segmentation process, the latter was much more efficient.

Regarding the other precollege predictor, the relationship between ACT scores and graduation rate is weak, which can easily be seen when the study sample is divided arbitrarily into three groups. Graduation rates for the middle (ACT from 15 to 19) and highest ACT subgroups were 41.4% and 51.9%, respectively, while the graduation rate for the lowest subgroup was only 28.0% (Table 2); these subgroups differed insignificantly (p = .210 for a likelihood ratio chi square). Logistic regression of graduation on ACT scores was also not significant at the .05 level (Table 3); ACT scores correctly predicted only 61.2% of third-year graduation (compared to 51.4% by chance and 41.0% actually observed). In segmentation modeling, ACT scores predicted the same as with logistic regression (Table 3). Segmentation showed that 52.5% of the 59 students with an ACT score over 17 graduated in three years and only 32.5% graduated if their ACT score was 17 or less. In general, however, ACT scores alone were not effective for predicting the likelihood of graduation, an observation that confirms the findings of Mouw and Khanna (1993).

Regarding an early college predictor, the relationship of first-quarter GPA and graduation can easily be seen when the study sample is divided arbitrarily into three subgroups based on grouped GPA (Table 2). The resulting subgroups differed substantially with regard to graduation (p < .01) and illustrate a positive correlation. The graduation rate for the upper (B and A) subgroup was 5.6 and 2.3 times more than the lowest (F and D) and middle (C) subgroups, respectively. In logistic regression, the first-quarter GPA correctly predicted 66.2% of the students who graduated by the third year (compared to 47.1% by chance and the observed 41.0%) (Table 3) and accounted for 22.8% of the variation (as indicated by the Nagelkerke [R.sup.2] in Table 3). In segmentation modeling, first-quarter GPA predicted correctly for 66.9% of the students (as compared to 48.3% by chance) (Table 3). It is clear that the first-quarter GPA can give a good indication as to which students are most likely to graduate. For example, only 12.5% of the 48 students with a first-quarter GPA of 2.24 or less graduated, but 56.0% of the 91 students with a first-quarter GPA over 2.24 (including 75.0% of the 28 students with a first-quarter GPA over 3.23) graduated in three years. Segmentation modeling displayed a minor nonlinear relationship when the GPA of the first quarter was less than or equal to 2.24; in this case, 40% of 10 students graduated if the GPA was less than 1.56, but only 8% of 26 did if the first-quarter GPA was between 1.56 and 2.24.

The final grade in the required, five-week orientation course has been another sound, early-college predictor of persistence and graduation (Zimmerman, 2000) at OSU-ATI. Hyers and Joslin (1998) also show how such grades are general indicators of persistence and graduation at their four-year, liberal arts college, and considerable work by others documents the positive role that such courses have (e.g., Barefoot, Warnock, Dickinson, Richardson, & Roberts, 1998). The relationship of orientation grade and graduation can easily be seen when the study sample is divided arbitrarily into three groups. The two highest subgroups (all three B and both A grade categories) differed substantially from the lowest one (C+ and below) with regard to graduation, illustrating a positive correlation (Table 2). The graduation rates for the two highest subgroups were 157% and 154% more than the lowest subgroup, respectively.

In logistic regression, orientation grade correctly predicted 67.6% of third-year graduation (compared to 50.5% by chance and the 41.0% observed) (Table 3) and accounted for 25.5% of the variation (as shown by the Nagelkerke [R.sup.2] in Table 3). Orientation grade was not a significant predictor of graduation in logistic regression with this sample size due in part to it being defined as a categorical variable with 12 values; but, if treated as a continuous variable, orientation grade was significant in this and most of the models below (and with about the same predictive ability). In segmentation modeling, orientation grade predicted correctly for 64.0% of the students (as opposed to 48.3% by chance) (Table 3). The segmentation program combined similar groups (based on orientation grade) and created two segments with the division between grades of B- and B. Graduation rates for the lower (40% of the sample) and upper subgroups (60% of the sample) were 21.4% and 54.2%, respectively.

Multivariate Patterns

The foregoing discussion focused on variations in graduation rate as influenced by single variables in isolation. It revealed that differences among subgroups varied about as expected with differences in high school rank, ACT scores, first-quarter GPA, and orientation grade; regression techniques and segmentation modeling compared favorably as well (Table 3). Comparing all or some of these predictors simultaneously, however, may have better explanatory power and may help identify distinctive subgroups of students. Only 41% of the study sample of 139 students graduated in 3 years or less, but segmentation modeling identified multivariate subgroups with both much better and far worse graduation success. The models that are discussed below are only examples; other ones could be developed, but these illustrate the comparative benefits of logistic and segmentation modeling. Owing to space limitations caused by the physical size and complexity of multivariate decision trees, only two relatively simple ones can be illustrated in this paper (Figures 2 and 3).

Both logistic regression and segmentation modeling had about the same overall predictive success when using the precollege ability surrogates of high school rank and ACT scores (Table 4), although ACT was not a significant predictor with high school rank in the logistic model. The best graduation rate (60.9% graduated) was for the subgroup with high school ranks over 60 (33% of the sample), but only 31.2% of the students graduated if high school rank was less than 60 (Figure 2). Of the latter group with high school rank less than or equal to 60 (67% of the sample), 45.7% of the students graduated if their ACT was greater than 17, but only 22.4% did if their ACT scores were less than or equal to 17 (41.7% of the sample). Although statistically significant subsidiary subgroups were generated in this model, the sample sizes were small. However, the model confirmed that students with combinations of low high school rank and low ACT scores were very unlikely to graduate (e.g., none of the 10 students graduated in a segment with high school ranks less than or equal to 20 and ACT scores less than or equal to 17). But if high school ranks were 21-60 while ACT scores were less than or equal to 17, then 27.1% graduated (13 of 48 students). This model shows the same nonlinear relationship of high school rank and graduation as illustrated above (graduation rates were much higher for ranks of 61-76 than for ranks of greater than 76).

Using the early college orientation grade variable with the precollege variables of high school rank or ACT scores in a multivariate analysis improved prediction slightly compared to that of the latter variables alone. In logistic regression, the overall models were significant, but Wald statistics showed that individual predictors (orientation grade used with high school rank and both ACT and orientation grade when in the same model) were not, thus raising questions about the models (Table 4). On the other hand, segmentation modeling created relatively homogeneous subgroups with these variables. For example, using orientation grade and high school rank, 20 of 23 (87.0%) students with an orientation grade of B or B+ and rank from 48-80 graduated in three years, while 69.6% (32 of 46 students) graduated if orientation grade was B or higher and rank was 48 or more. Reorganizing the predictors in this model revealed other useful segments. For example, when rank was the first predictor and was 60 or less, only 31.2% graduated. But within this segment were diverse subsidiary ones. For example, 52.9% graduated if they had a B+, A-, or A for orientation grade; however, of this subgroup, 37.5% (9 of 24 students) graduated if rank was from 47-60 (as compared to the 90.0% rate for the segment with ranks of more than 60).

Using orientation grade and ACT scores together in segmentation modeling showed the positive effects of each. If the orientation grade was higher than B-, 54.2% of 83 students graduated, but if the ACT scores for students in this group were higher than 17, 61.5% of 39 graduated, and only 47.7% (21 of 41) graduated if ACT scores were 17 or less. Also, if the orientation grade was B- or lower while the ACT score was 16 or lower, only 8.0% (2 of 25) graduated. A combination of low ACT score and low orientation grade is clearly detrimental to graduation.

Using the early college, first-quarter GPA variable with the precollege variables increased predictive success. High school rank and first quarter GPA together predicted well overall (about 72% to 75%)in both logistic regression and segmentation modeling (Table 4). Both variables generally correlated positively with graduation, as would be expected (and as the discussion above confirmed) and which segmentation modeling (Figure 3) illustrates well. For example, if first-quarter GPA was over 2.24, 56.0% of 91 students graduated but just 6 of 48 (12.5%) did if first-quarter GPA was 2.24 or lower. However, 69.2% of 52 in this segment with first-quarter GPA greater than 2.24 graduated if their high school rank was over 47. If first-quarter GPA exceeded 3.15 while high school rank was 47-80, then 14 of 15 (93.3%) graduated. But segmentation modeling also revealed nonlinear relationships with these two predictors. For example, if first-quarter GPA was more than 2.24 while high school rank was more than 80, then only 4 of 10 graduated, which was far fewer than the 76.2% (32 of 42 students) in the segment with the same first-quarter GPA (greater than 2.24) but lower high school ranks (from 48-80).

This model shows well the negative effects of low first-quarter achievement even for students with high school ranks near and above the median value, thereby reinforcing the belief that early college achievement is a strong indicator of student intent. In fact, in the segment of Figure 3 with first-quarter GPA less than or equal to 2.24, in a lower level (omitted to save space and since only 6 of its 48 students graduated) with first-quarter GPA from 1.56 to 2.24, only 2 of 14 students (14.3%) with a high school rank of 49 or higher graduated (their mean rank was 62.7 and the highest was 85).

Using only the two early college variables improved prediction only slightly with respect to the multivariate models already discussed (Table 4). A logistic regression model with first-quarter GPA and orientation grade successfully predicted 75.5% of cases, while segmentation modeling correctly predicted 76.3%. As the discussion concerning single predictors illustrated, graduation rates were better with higher values of both variables. Nonlinear relationships, however, were apparent with segmentation modeling. For example, in a subgroup with first-quarter GPA less than or equal to 2.24 but less than or equal to 3.265, students with an orientation grade of A- or A graduated at lower rates (26.7% of 15 students) than students with an orientation grade of B+ or lower (54.2% of 48). It is interesting to note here that, in a multivariate segmentation model, the two precollege variables predicted nearly the same as a model with only the two early college variables (71.2% and 76.3%, respectively). This observation suggests that waiting for postadmission achievement data to predict graduation may not be necessary.

Table 5 summarizes four multivariate logistic and segmentation models with three or more predictors. Each logistic model has one or more independent variables with nonsignificant Wald statistics. Only two of these models are discussed here for illustrative purposes, but generally they do not yield much more useful information than the two-variable models of Table 4, due mainly to the small sample size used in this study. Models using more than two predictors can be powerful but are less useful for small sample sizes because segments tend to be small below the second level of the classification tree.

Multivariate models utilizing three or all four predictor variables, however, provided additional insight. Using orientation grade, high school rank, and first-quarter GPA together in the same logistic or segmentation model yielded high overall predictive success (77.7% and 79.9%, respectively, although the orientation grade and high school rank were insignificant in logistic regression). In general, high values of these three variables were associated with high graduation rates, but a nonlinear relationship was also apparent in segmentation modeling. The best rate of 100% was for a very small sample of 5 students who had a high school rank of less than 48, an orientation grade of B or higher, and a first-quarter GPA from 2.51 to 2.85. The next best rate of 93.3% was for the 14 of 15 students who had a first-quarter GPA over 3.15 and high school ranks from 48 to 80. However, the model also showed that if first-quarter GPA was more than 2.24 but less than or equal to 3.15, lower orientation grades were associated with higher graduation rates: 76.2% of 21 students with orientation grade of B+ or lower graduated but only two of six students did if the orientation grade was A- or A. The worst rate (0 of 13 students) characterized students in the subgroup with a first-quarter GPA of over 1.785 but less than or equal to 2.24 while orientation grade was B- or less (and only 4 of 42 students graduated if orientation grade was B- or less while the first quarter GPA was less than or equal to 2.24). In spite of some minor nonlinear relationships, this model shows the strong positive relationships involving these three variables and reinforces the idea that these early college and precollege variables are important predictors which could be employed usefully in strategies to improve graduation rates.

Finally, a segmentation model that used four variables simultaneously (high school rank, ACT, first-quarter GPA, and orientation grade) was useful but complex. The overall predictive ability of both logistic regression and segmentation modeling (Table 5) was 77-81%, but 3 of the predictors in logistic regression (high school rank, ACT, and orientation grade) had insignificant Wald statistics. The best segmentation model excluded the ACT variable, but if high school rank was selected manually as the first predictor, all 4 independent variables defined the decision tree. In this case, 100% of 16 students in one subgroup graduated if (a) their high school rank was in the range of 60-80, (b) their first-quarter GPA was more than 2.195, and (c) their orientation grade was B+ or less (but only 63.6% of 11 students graduated if their orientation grade was A- or A). The next best subgroup (73.9% of 23 students) was for students with (a) intermediate values of high school rank (26-60), (b) orientation grades of B or better, and (c) first-quarter GPA over 2.505. At the other extreme, only 3 of 37 students (8.1%) graduated if they were in a subgroup with (a) mid to low high school rank (60 or less), mid to low first-quarter GPA (less than 2.505), and mid to low ACT scores (less than 18). This model shows relationships that would not be readily apparent with logistic regression.

Conclusion

In this analysis of three-year graduation rates, a major focus was the comparison of logistic regression and segmentation modeling. The latter is useful in spite of a small sample size because it visually depicts clear subgroups that can be analyzed further and which illustrate basic tendencies in the data. Although logistic regression is a useful and standard technique, it cannot as easily reveal the same relationships, and it cannot illustrate the nonlinear ones. For example, logistic regression could not illustrate the negative correlations for specific ranges of values within an overall generally positive relationship, such as with high school rank and first-quarter GPA, in both univariate and multivariate models. Both logistic regression and segmentation modeling have distinct advantages over simple correlations (point biserial, in this case) or crosstabulations because they both can consider several variables simultaneously. Segmentation modeling would be useful to admission officers, counselors, and faculty because it identifies clearly defined subgroups that can be targeted for appropriate action (depending on the predictor variables), such as which applicants to admit (or not admit) or which students might benefit from proactive advising.

Higher graduation rates generally occurred for subgroups with higher values of all the independent variables used in this study (i.e., ACT scores, high school rank, orientation grade, and the first-quarter GPA). A generally positive relationship was clearly evident with a simple crosstabulation, and logistic regression coefficients for these independent variables in both univariate and multivariate models were all positive. These general observations would be expected logically and have been recognized for a long period of time.

A completely positive relationship, however, was not always true with the closer scrutiny provided by segmentation modeling; the finding of nonlinear patterns with this method is a new observation about these traditional predictors. For example, graduation rates were best in the mid to high (but not highest) range of high school ranks (about 60-76). Also, relatively high graduation rates characterized some students with mid to low values of high school rank and mid to high ACT scores. Finally, relatively high graduation rates characterized the subgroup of students who had an orientation grade of B or B+ and mid to high (but not the highest) values of high school rank. Segmentation modeling also revealed nonlinear relationships for some of the lower values of these predictors. For example, relatively low graduation rates typified students who had a first-quarter GPA in the C- to C+ range, or who had both mid to low high school ranks and ACT scores. Segmentation modeling was also useful for showing some synergistic-like effects; for example, low ACT scores coupled with low high school ranks, or orientation grades of B- or less accompanied by mid to low ACT scores.

Being able to identify subgroups and illustrate how well students in them graduate is a major advantage of segmentation modeling. Although some of these subgroups were small, the nonlinear relationships should be investigated further because they might suggest underlying trends or identify unexpected problems. For example, a surprising finding was that students with the best background did not necessarily graduate at the best rates (such as with high school ranks). The fact that some students with the best academic background have less than expected graduation rates may reveal the effects of personal motivation, reasons for enrolling, or other influences. An advantage of the segmentation process is that it may identify subgroups that could benefit from intervention strategies to improve retention and graduation rates.

This study also clearly illustrates the ability of the grade in a five-week orientation course to point to student intentions (Zimmerman, 2000). This type of variable is not traditionally included as a predictor of graduation; however, its ability to predict graduation rivals that of traditional variables. This orientation grade clearly identifies students who might benefit from proactive advising and is especially useful because it is available before the end of the important first quarter. The relationship is positive and mainly linear, but for some reason, students earning an orientation grade of A- graduated at lower rates (35.3% of 17) than students who earned an A (66.7% of 18) or B+ (68.2% of 22). In fact, all the minus grades (A-, B-, C-) had relatively low graduation rates. Although this univariate relationship was also apparent from simple crosstabulation, exploratory segmentation modeling was the first to point out this observation.

The findings reported in this article are based solely on full-time, traditional, A.A.S., terminal-degree students at one college. The results may not be representative of student populations at other colleges. Further studies should investigate the power of segmentation modeling in institutional research to address retention, graduation, and other related student achievement issues. The method is easy to apply using common software and the results are statistically sound enough to satisfy researchers and understandable enough for policy makers.

APPENDIX I

Table 1
Summary statistics for predictor variables

                              Study Sample         Population
Source      Variable
                           Mean   Median    N   Mean   Median   N

            ACT composite  17.26  17.00    139  17.22   17.00   153
Precollege
            High school
             rank          50.51  50.00    139  50.21   49.00   142

            GPA of 1st
Early        quarter        2.45   2.61    139   2.45    2.59   160
college     Orientation
             grade          2.74   3.00    139   2.72    3.00   160

NOTE: The population includes all students who had an opportunity
to graduate in three years. These students enrolled in the seven
cohorts from 1992-1998. The study sample includes only students
who had both a valid ACT score and a valid high school rank.
Table 2
Comparison of graduation rates (in three
years) by categories of predictors

                                                   Graduation in
                                                    three years
Source       Variable     Categories     Limits
                                                   Graduation in
                                                   subgroup (%)

                             Low           34          25.7
            High school  Intermediate    35-65         35.8
               rank          High          66          59.0
Precollege
                             Low           14          28.0
                ACT      Intermediate    15-19         41.4
             composite       High          20          51.9

                             Low          F, D         10.5
            GPA of 1st   Intermediate      C           25.5
              quarter        High         B, A         58.9
Early
college                      Low        F, D, C        20.0
            Orientation  Intermediate      B           50.8
               grade         High          A           51.4

                                      Graduation in three years

Source       Variable     Categories
                                          Number    Subgroup
                                        graduated       N

                             Low            9          35
            High school  Intermediate       25         65
               rank          High           23         39
Precollege
                             Low            7          25
                ACT      Intermediate       36         87
             composite       High           14         27

                             Low             2         19
            GPA of 1st   Intermediate       12         47
              quarter        High           43         73
Early
college                      Low             9         45
            Orientation  Intermediate       30         59
               grade         High           18         35

                                           Difference among
                                              subgroups

Source       Variable     Categories   Likelihood
                                       ratio chi-   df      P
                                         square

                             Low
            High school  Intermediate    8.852       2    .012
               rank          High
Precollege
                             Low
                ACT      Intermediate    3.125       2    .210
             composite       High

                             Low
            GPA of 1st   Intermediate    23.113      2   <.001
              quarter        High
Early
college                      Low
            Orientation  Intermediate    12.872      2    .002
               grade         High
Table 3
Logistic and segmentation models for
single variable predictors of graduation

                                              Logistic regression

Source       Predictor                           Wald statistic
             variables               Model

                            chi-      df     p    chi-   df     p
                           square                square

                ACT
             composite      3.53       1    .06   3.42    1    .06
Precollege
               High
              school
               rank         8.74       1   <.01   8.17    1   <.01

            GPA of 1st
              quarter      25.78       1   <.01  19.00    1   <.01
Early
college     Orientation
               grade       29.11      10   <.01  13.46   10    .20 *

                         Logistic                       Segmentation
                         regression                       modeling

Source       Predictor               Prediction (%)   Prediction (%)
             variables
                         [R.sup.2]  Overall  Chance   Overall  Chance
                                    correct  correct  correct  correct

                ACT
             composite     .034      61.2     51.4     61.2     51.4
Precollege
               High
              school
               rank        .082      65.5     52.1     69.1     55.9

            GPA of 1st
Early         quarter      .228      66.2     47.1     66.9     48.3
college
            Orientation
               grade       .255      67.6     50.5     64.0     48.3

NOTE: Chi-square is a chi-square statistic, df is degrees of freedom,
and p is significance level. [R.sup.2] is the Nagelkerke [R.sup.2] a
statistic that attempts to quantify the proportion of "explained"
variation in the logistic regression models; it is analogous to the
[R.sup.2] of linear regression. The chance correct prediction in both
logistic and segmentation modeling is based on marginal totals.
Orientation grade is a categorical variable with 12 categories (F, D-,
D, D+, C-, C, C+, B-, B, B+, A-, A). An asterisk (*) indicates that
orientation grade is a significant predictor (p <.01) if treated as a
continuous variable.
Table 4
Logistic and segmentation models for
two-variable predictors of graduation

Source          Predictor                 Model
                variables
                                 chi-      df        p
                                square

Precollege   High school rank
only          ACT composite       9.77      2      <.01

             High school rank
            Orientation grade    34.08     11      <.01
Mixed
precollege    ACT composite
and early   Orientation grade    30.81     11      <.01
college
             High school rank
            GPA of 1st quarter   28.88      2      <.01

Early       GPA of 1st quarter
college     Orientation grade    37.33     11      <.01
only

                                  Logistic regression

Source          Predictor         Wald statistic
                variables
                                   chi-       df    p      [R.sup.2]
                                  square

Precollege   High school rank      5.97        1   .01
only          ACT composite        1.01        1   .31       .091

             High school rank      4.76        1   .03       .293
            Orientation grade     12.04       10   .28 *
Mixed
precollege    ACT composite        1.64        1   .20       .268
and early   Orientation grade     12.50       10   .25 *
college
             High school rank      3.04        1   .08       .253
            GPA of 1st quarter    15.53        1  <.01

Early       GPA of 1st quarter     7.21        1  <.01       .318
college     Orientation grade      7.57       10   .67
only

                                                     Segmentation
                                                       modeling
Source          Predictor
                variables        Prediction (%)    Prediction (%)

                                Overall  Chance   Overall  Chance
                                correct  correct  correct  correct

Precollege   High school rank
only          ACT composite      64.7     49.9     71.2     53.7

             High school rank
            Orientation grade    75.5     52.4     73.4     54.3
Mixed
precollege    ACT composite
and early   Orientation grade    69.1     51.5     66.9     50.1
college
             High school rank
            GPA of 1st quarter   71.9     50.7     74.8     53.6

Early       GPA of 1st quarter
college     Orientation grade    75.5     49.6     76.3     52.0
only

NOTE: Chi-square is a chi-square statistic, df is degrees of freedom,
and p is significance level. [R.sup.2] is the Nagelkerke [R.sup.2],
a statistic that attempts to quantify the proportion of "explained"
variation in the logistic regression models; it is analogous to the
[R.sup.2] of linear regression. The chance correct prediction in both
logistic and segmentation modeling is based on marginal totals.
Orientation grade is a categorical variable with 12 categories (F, D-,
D, D+, C-, C, C+, B-, B, B+, A-, A). An asterisk (*) indicates that
orientation grade is a significant predictor (p <.01) if treated as a
continuous variable.
Table 5
Logistic and segmentation models for three- and four-variable
predictors of graduation using combinations of precollege and
early college predictors

                                            Logistic regression

Predictor variables           Model         Wald statistic

                      chi-                 chi-
                     square   df    p     square   df    p

High school rank                            2.71    1    .10
GPA of 1st quarter   28.96    3    <.01    14.77    1   <.01
ACT composite                               0.09    1    .77

High school rank                            2.58    1    .11
Orientation grade    39.98    12   <.01     7.37   10    .69
GPA of 1st quarter                          5.34    1    .02

High school rank                            3.73    1    .05
ACT composite        34.65    12   <.01     0.56    1    .46
Orientation grade                          11.60   10    .31 *

High school rank                            2.39    1    .12
GPA of 1st quarter                          4.84    1    .03
ACT composite        40.00    13   <.01     0.02    1    .89
Orientation grade                           7.29   10    .70

                                                   Segmentation
                                                     modeling

Predictor variables              Prediction (%)    Prediction (%)

                     [R.sup.2]  Overall  Chance   Overall  Chance
                                correct  correct  correct  correct

High school rank
GPA of 1st quarter     .254      71.2     50.6     80.6     52.8
ACT composite

High school rank
Orientation grade      .337      77.7     49.4     79.9     53.7
GPA of 1st quarter

High school rank
ACT composite          .297      72.7     50.8     74.8     53.3
Orientation grade

High school rank
GPA of 1st quarter     .337      77.0     49.3     80.6     52.5
ACT composite
Orientation grade

NOTE: Chi-square is a chi-square statistic, df is degrees of freedom,
and p is significance level. [R.sup.2] is the Nagelkerke [R.sup.2], a
statistic that attempts to quantify the proportion of "explained"
variation in the logistic regression models; it is analogous to the
[R.sup.2] of linear regression. The chance correct prediction in both
logistic and segmentation modeling is based on marginal totals.
Orientation grade is a categorical variable with 12 categories (F, D-,
D, D+, C-, C, C+, B-, B, B+, A-, A). An asterisk (*) indicates that
orientation grade is a significant predictor (p < .01) if treated as a
continuous variable.

References

ACT. (2001). National Dropout and Graduation Report [On-line]. Available: http:/ /www.act.org/news/releases/2001/update.html/11-27-01.

Barefoot, B. O., Warnock, C. L., Dickinson, M. P., Richardson, S. E., & Roberts, M. R. (Eds). (1998). Exploring the evidence: Reporting outcomes of first-year seminars, National Resource Center for the First-Year Experience and Students in Transition, University of South Carolina, Volume II.

Baur, R., & Kreps, G. (1996). Using TQM for retention enhancement. NACTA Journal, 40 (4), 22-26.

Biggs, D., de Ville, B., & Suen, E. (1991). A method of choosing multiway partitions for classification and decision trees. Journal of Applied Statistics, 18, 49-62.

Breiman, L. Freidman, J. H., Olshen, R.A., & Stone, C.J. (1984). Classification and regression trees. Belmont, CA: Wadsworth.

Hosner, D. W., & Lemeshow, S. (1989). Applied logistic regression. New York: John Wiley and Sons, Inc.

Hyers, A. D. (2001). Predictable achievement patterns for student journals in introductory earth science courses. Journal of Geography in Higher Education, 25, 53-66.

Hyers, A. D., & Anderson, P. S. (2001). Interpreting variability of examination scores for geography education. National Council for Geographic Education, 86th Annual Meeting. University of British Columbia, Vancouver, BC.

Hyers, A. D., & Joslin, M. (2001, April 10). Academic issues and retention. Unpublished manuscript, Massachusetts College of Liberal Arts.

Hyers, A. D., & Joslin, M. (1998). The first year seminar as a predictor of academic achievement and persistence. Journal of the Freshman Year Experience, 10 (1), 7-29.

Kalsbeek, D. (1987). Campus retention: The MBTI in institutional self-studies. In J. Provost & S. Anchors (Eds.), Applications of the Myers-Briggs Type Indicator in higher education (pp. 31-64). Palo Alto, CA: Consulting Psychologists Press.

Loh, W.Y., & Shih, Y.S. (1997). Split selection methods for classification trees. Statistica Sinica, 7, 815-840.

Mouw, J., & Khanna, R. (1993). Prediction of academic success: a review of the literature and some recommendations. College Student Journal, 27 (3), 328-336.

Noel, L., Levitz, R., & Saluri, D. (1985). Increasing student retention: New challenges and potential. San Francisco: Jossey-Bass.

Press, S.J., & Wilson, S. (1978). Choosing between logistic regression and discriminant analysis. Journal of the American Statistical Association, 73, 699-705.

Reisberg, L. (1999). Colleges struggle to keep would-be dropouts enrolled. The Chronicle of Higher Education, 46 (7), A54-A56.

Schurr, K., Ruble, V., Palomba, C., Pickerill, B., & Moore, D. (1997). Relationships between the MBTI and selected aspects of Tinto's model for college attrition. Journal of Psychological Type, 40, 31-42.

SPSS. (1998). AnswerTree 2.0 User's Guide. Chicago: SPSS, Inc.

Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition. Chicago: University of Chicago Press.

Tinto, V. (1975). Dropouts from higher education's theoretical synthesis of recent research. Review of Educational Research, 45, 89-125.

Wilkie, C., & Redondo, B. (1996). Predictors of academic success and failure of first-year students. Journal of the Freshman Year Experience, 8 (2),17-32.

Zimmerman, A. (2000). A journal-based orientation course as a predictor of student success at a public, two-year, technical college. Journal of the Freshman Year Experience, 12 (1), 29-43.

Albert D, Hyers is a professor of geography at Massachusetts College of Liberal Arts in North Adams, Massachusetts. [e-mail]

Allen Zimmerman is a professor of engineering technology and technical physics at The Ohio State University, Wooster Campus in Wooster, Ohio. [e-mail]