Measuring the effectiveness of developmental writing courses.

The community college is the only academic safety net for underprepared students seeking a college education, and the number of developmental students continues to grow for several reasons. Social pressures, such as reforms limiting entitlements over an individual's lifetime, have led more welfare recipients into the classroom as they prepare for jobs that pay a living wage (Jenkins & Boswell, 2002). Refugees and immigrants want the opportunities a college degree brings, but they may require developmental work (Jenkins & Boswell, 2002). Many native-born U.S. students also require remediation; almost two-thirds of high school students in Florida require developmental courses before they can attempt college-level work (Office of Educational Services and Research, Division of Community Colleges, 1999). Despite these pressures, states are increasingly mandating that four-year schools cannot offer developmental classes (Jenkins & Boswell, 2002); thus, if developmental students fail at the community college, they do not have access to other academic institutions. Therefore, effective remedial courses at community colleges are crucial to developmental students' future academic success. One proof of effectiveness is accurate placement into developmental courses; another is success in subsequent courses (Schmitz & delMas, 1991; Weissman, Bulakowski, & Jumisko, 1997).

As institutions that welcome these underprepared students, community colleges have an obligation to equip their students with the skills needed to succeed in college courses. Like its counterparts, Okaloosa-Walton Community College (OWCC) in northwest Florida offers developmental courses in reading, writing, and math to fulfill its primary mission of teaching all students. A periodic assessment of the effectiveness of these courses is essential to ensure that developmental students are being adequately served. Effectiveness cannot be assumed; in 1998, more than a quarter of developmental students nationwide failed to complete their college preparatory coursework (Jenkins & Boswell, 2002).

Faculty and administrators at OWCC needed to know if the college's developmental writing course, College Prep English II, was preparing students for success in Composition I (a college-level course). As English teachers, the researchers wanted to know whether the state-mandated multiple-choice test used to place students in developmental English provided accurate placement. Therefore, in this study, the researchers assessed the effectiveness of College Prep English II in preparing students for success in Composition I and investigated the relationship between scores on the Florida College Placement Test (FCPT) and success in developmental English and other writing-intensive courses.

Review of the Literature

Studies on the effectiveness of basic writing programs have concluded overall that these programs are effective. Three such studies, by Stein (1982) at Minnesota Community College, by Ragland (1997) at Central Missouri State University, and by Weissman et al. (1997) at the College of Lake County (a community college in Chicago), concluded that students who completed developmental skills courses were more likely to succeed in college-level writing classes than were students who did not complete preparatory work. Glau (1996) reported that developmental courses that focused on grammar at Arizona State University were ineffective but that an extended developmental program that concentrated on writing did prepare students for success in Composition I.

Researchers have also concluded that developmental courses should be required and not optional when students fall below the cut-off score on placement exams (Berger, 1997; Weissman et al., 1997). However, placement tests can be imperfect tools, even when content validity has been achieved (Schmitz & delMas, 1991). Schmitz and delMas (1991) noted that while a test's content validity is essential, decisions based on the test must also prove to be accurate. If a test facilitates accurate decisions about placement, students who score low but do not take developmental classes will not do as well academically as their counterparts who receive remediation, and retention will be higher for students who need and receive preparatory work before college classes (Schmitz & delMas, 1991). McCormick and McCormick's 1986 study of students in developmental writing classes at Eastern Illinois University addressed the correlation between the students' placement scores and their future academic success in all their other classes. Comparing developmental students to non-developmental students, the researchers found no significant difference in the areas of credits earned, graduation rates, and grades earned in college-level writing courses. However, those developmental students with lower placement scores had lower GPAs and were more likely to be placed on academic probation than were students with higher placement scores.

Purpose of the Study

The researchers wanted to evaluate OWCC's effectiveness in preparing developmental writing students to succeed in college-level English courses, and they were interested in whether the mandatory placement test's assignment of students in developmental courses was accurate. Thus, this study began with two research questions. First, is College Prep English II effective in redressing developmental students' deficits when they are faced with the academic demands of Composition I? Second, do students' scores on the FCPT relate positively to their success in writing-intensive courses?

The Courses

At Florida community colleges, students are placed into developmental courses by their score on the FCPT. At OWCC, a developmental writing course, College Prep English II, is required for students scoring between 69 and 82 on the sentence skills portion of this state-mandated placement test. As well as teaching grammar, College Prep English II teaches writing skills, including sentence structure, paragraph construction, coherence, revision, and editing. The developmental sequence may begin with College Prep English I if a student scores below 69 on the placement test, but all developmental students must take College Prep English II. The college credit sequence begins with an elective, Writing and Grammar, but students may and often do choose to enter Composition I directly from College Prep English II. Composition I focuses on organizing information and ideas in written outlines, communicating effectively in writing, locating research sources and documenting them in MLA format, and writing and passing a timed essay. At OWCC, all writing courses, including developmental courses, are limited to 22 students, and students must earn a C in order to pass these courses.

A Florida statute requires that college students write across the curriculum, although the implementation of this requirement is largely left to individual institutions. During the span of this study, OWCC students wrote 6,000 words in Composition I, in Composition II, and in other courses in the humanities, for a total of 24,000 words per student. For part of this study, we looked at all these writing-intensive courses.

The Placement Test

According to the OWCC test coordinator, the FCPT, designed by The College Board, consists of 35 questions of two different types. The first type requires students to choose the correct word or phrase to substitute for the underlined incorrect part of a sentence, and the second type requires that students choose a version of a sentence rewritten according to specific criteria while maintaining the same meaning as the original sentence. The FCPT tests the following skills: sentence logic, coordination and subordination, and recognizing complete sentences (J. M. Buchanan, personal communication, July 30, 2003). In its current coordinator guide, The College Board notes that students who score "about 86" on the sentence skills section of the test will be able to handle complex verb tenses and coordination and subordination, correct sentence fragments and misplaced modifiers, and resolve sentence problems that involve both grammar and logic (College Entrance Examination Board, 2002).

The Participants

In this study, the researchers examined the transcripts of 929 students. The participants were divided into four groups of students who were enrolled in OWCC's English course sequence between the fall 2000 and fall 2001 semesters. Students in Groups 1, 2, and 3 scored 82 or below on the FCPT and were therefore placed in College Prep English II or had been placed in and passed College Prep English I. The 58 students in Group 1 passed College Prep English II and enrolled in Composition I. The 48 students in Group 2 were also enrolled in College Prep English II, but although they were successful in College Prep English II, they did not subsequently enroll in Composition I. The 29 students in Group 3 were placed in College Prep English II, but these students failed the course. Groups 1, 2, and 3 were all enrolled in College Prep English II during the fall of 2000; thus, successful students had three more semesters in this study in which they could have enrolled in Composition I. For the purpose of this study, the researchers did not consider whether the participants had completed College Prep English I.

Group 4, consisting of 794 students, had never taken College Prep English II; these students' FCPT scores of 83 or above allowed them to enroll in Composition I without developmental course work. Group 4 students were enrolled in the same Composition I sections that Group 1 entered--thus Group 4 students took Composition I in any of the three semesters, spring 2001 through fall 2001, in which students in Groups 1, 2, and 3 enrolled in that course. In this way, the researchers hoped to eliminate the effect of the Composition I instructors in the analysis of College Prep English II's effectiveness in preparing developmental students for success in Composition I.

Procedure

To determine the effectiveness of College Prep English II in preparing developmental students for success in Composition I, the researchers compared the Composition I course grades of Groups 1 and 4 and ran a Pearson correlation between grades in College Prep English II and Composition I, setting a p value of <.05 to determine significance of the results. A frequency table was also constructed to analyze the grades earned by developmental students in both courses, and the percentage of students in each group who repeated Composition I was examined.

To determine the correlation of FCPT scores with student success in writing-intensive courses, the researchers performed three Pearson correlations. First, the researchers analyzed the College Prep English II grades and FCPT scores of Groups 1, 2, and 3. The researchers then analyzed the Composition I grades and FCPT scores for Groups 1 and 4. Lastly, the researchers looked for a correlation between students' grades in other English and writing-intensive humanities courses with their FCPT scores. With all Pearson correlations, a p value of <.05 was set to determine significance of the results.

Results

The Effectiveness of Developmental Writing

An analysis of the participants' transcripts yielded data concerning grades, attempts, and persistence. As shown in Table 1, there was no significant difference in the mean Composition I grades of the developmental (Group 1) and non-developmental (Group 4) students. The developmental students had better pass and retention rates than did their non-developmental counterparts.

In addition, when compared to a random sampling of 58 students from Group 4, the developmental students required fewer attempts at Composition I to pass. From Group 1 (who had taken College Prep English II) only 4 students, or 6.8%, had to repeat Composition I, whereas from the random sample of Group 4, seventeen students, of 29.3%, had to repeat Composition I.

A Pearson correlation found a relationship between Group 1 students' grades in College Prep English II and their grades in Composition I, but the correlation is too low to be meaningful (r(58) = .264, p =.045). A frequency table (Table 2) reveals that students earning As and Bs in College Prep English II usually find their grades dropping by at least a letter in Composition I.

The Correlation Between FCPT Sentence Skills Scores and Grades in Writing-Intensive Courses

The researchers next looked for a correlation between the FCPT scores and grades in the course in which the test placed students, beginning with those students placed in College Prep English II. For Groups 1, 2, and 3, all of whom scored at or below 82 on the FCPT, a Pearson correlation did not reveal a significant relationship between the FCPT sentence skills scores and College Prep English II grades (r(135) = .067, p = .455).

Then, correlations between students' success in other English courses and their FCPT scores were examined. First, the researchers looked at correlations between Composition I grades and FCPT scores. For Group 1 (who enrolled in Composition I after successfully completing College Prep English II) there was no significant correlation, and what correlation existed was negative (r(58) = -.068, p = .641). The researchers randomly sampled 58 students from Group 4 (whose test scores allowed them to go straight into Composition I without taking College Prep English II); of these students, only 33 had FCPT scores. (The SAT and ACT are also accepted as placement scores.) For these 33, the FCPT had no significant correlation with their Composition I grade (r(33)= .010, p =.957).

Next, the researchers looked for correlations between the FCPT and students' grades in other courses in the English sequence, namely, Writing and Grammar and Composition II. From Group 1, thirty-nine of the 58 students enrolled in one or both of these courses, and the FCPT had no significant correlation with those course grades (r(39)=. 186, p =. 191). From the random sample students from Group 4 (the non-developmental students), 33 of the 58 students in the random sample enrolled in one or both of these courses, but only 14 had FCPT scores. For these 14, the FCPT again had no significant correlation with grades in these courses (r(14)= .157, p = .0521).

Lastly, the researchers looked for correlations between the FCPT and writing-intensive courses outside the English course sequence. Some students in Groups 1 and 2 took these courses after passing College Prep English II. The correlation between these humanities grades and FCPT scores was not significant, and what minimal correlation existed was negative (r(11) = -.139, p = .411). The college prohibits developmental students who fail College Prep English II from attempting other writing intensive courses, but two students from Group 3 (who had failed the developmental course) managed to circumvent this rule; each took a writing-intensive humanities course. One student passed and one failed. Within the random sample of 58 non-developmental students (from Group 4), 44 students took these humanities courses, but only 21 had FCPT scores. For these 21, the FCPT had no significant correlation with course grades (r(21) = .071, p = .706).

Discussion

This study indicates that College Prep English II at OWCC prepares students to succeed in Composition I, and thus the findings confirm previous research. There was no significant difference in the mean Composition I grades of developmental and non-developmental students. In addition, the developmental students passed Composition I at a higher rate than that of non-developmental students, withdrew at a lower rate than their non-developmental counterparts, and required fewer attempts to pass.

However, although 74% of developmental students who enrolled in Composition I succeeded, the correlation between success in the two courses is weak. The researchers believe that a shift in grades between the two courses accounts for this low correlation between developmental students' success in College Prep English II and Composition I. A frequency table reveals that successful developmental students typically see their grade in Composition I drop by at least a letter below the grade they received in College Prep English II. This drop in grades may be explained by the significant gap that exists between the content of the two courses: While College Prep English II students are assessed on sentence skills and paragraphs, Composition I students are assessed on essays, at least one of these being a long research paper. Thus, writing style, research skills, and competency in documentation are the assessment criteria in Composition I--a substantial leap from the sentence skills and paragraph evaluation criteria of College Prep English II. In addition, Composition I is more demanding simply in terms of workload; students must write a minimum of 6,000 words.

The study also produced an ambiguous and potentially disturbing finding: the lack of a significant correlation between state-mandated placement test scores and grades in all writing-intensive courses. We had expected that students whose FCPT scores were closer to the cut-off score of 82 would do better in College Prep English II than those scoring far below the cut-off score because their deficits in writing skills were not as large as that of low-scoring students. However, the low correlation between the test scores and grades indicates that the size of the deficit, as measured by the FCPT, does not affect students' grades. Not only did College Prep English II and Composition I grades fail to show a correlation with the FCPT, but grades in all writing-intensive courses failed to show a correlation with the test as well. Three interpretations of this finding are possible (Schmitz & delMas, 1991). First, and most optimistically, this low correlation may mean that the placement test is accurate, and instruction targets students' areas of weakness. Schmitz and delMas (1991) note that the correlation between placement scores and test grades may be very low when placement is precise: If instruction meets developmental students' needs, then their low placement scores may be followed by high grades in the course in which the test placed them.

Unfortunately, this explanation is not convincing in the case of the FCPT because of the test's apparent lack of alignment with College Prep English II. Based on information provided by The College Board (the test developer), the researchers conclude that the FCPT lacks content validity as a placement instrument for the course. The FCPT tests students' skills in coordination and subordination, logic, and misplaced modifiers, none of these concepts are covered in the developmental English course. The only FCPT skills that are taught in College Prep English II are correcting verb tenses and fragments. If the test is not aligned with the course in which it places students, one would expect little or no correlation between the FCPT and students' grades (Schmitz & delMas, 1991).

Schmitz and delMas (1991) provide a third explanation of a low correlation between test scores and course grades: an invalid use of the test, such as using a cut-off score that is not the optimum. While we believe that the primary reason for the lack of a significant correlation is with the test's content validity, an inappropriate cut-off score or reliance solely on this instrument could also contribute to the statistical findings. In fact, The College Board recommends that placement should depend on multiple factors, not just on its test results. White (1998), who recommends holistic scoring of multiple writing samples as the preferred method of writing assessment, notes that multiple-choice tests purporting to measure writing skills often bear little relationship to students' real writing ability. If multiple-choice tests must be used, White urges, they should be combined with an essay test. Wolcott and Legg (1998) observe that there is a significant difference between students' ability to identify an error and choose a corrected sentence and their ability to produce a correct sentence in their own writing. In College Prep English II, students are assessed on their writing skills--their creation of sentences and paragraphs; on the FCPT, they are assessed on their recognition of errors and selection of corrections created by someone else.

The researchers' impression is that the test does place students who need remediation in writing in College Prep English II. One researcher (Southard) taught college preparatory English for 8 years and informally coordinated those courses for 4 of those years, and her experience showed that the students in College Prep English II really do require a developmental course. In addition, other college preparatory instructors strongly declare their students to be unprepared for college-level writing. However, the test's placement of developmental students is not necessarily accurate. Approximately one third of incoming students will need remediation, so the odds are good that any test of general language skills will identify many students whose writing skills require developmental coursework. But the test may not place all students precisely.

Instead, the researchers believe the FCPT identifies some students who need remediation but not all who could benefit from developmental work. The students in this study who passed the FCPT, and thus should not have needed developmental work, failed Composition I at a rate of 37%. Of course, there are many reasons students fail a course, but when coupled with the FCPT's low content validity for both College Prep English II and Composition I, this failure rate adds to concerns about the test's accuracy.

This study identifies two areas for future research. More research on the correlation of the FCPT with the skills taught in developmental English courses is urgently needed because this state-mandated test is used as a primary placement tool in post-secondary institutions across Florida. Similarly, other placement tools used in other states should be evaluated for their content validity and for their suitability as the basis for placement decisions.

In addition, the apparent gap between developmental English and Composition I needs further study. The low correlation between grades in College Prep English II and Composition I and the drop in developmental students' grades from College Prep English II to Composition I suggest a gap between a successful developmental student's writing skills and the necessary entry-level skills for a passing Composition I student.

Faculty in every developmental writing program should periodically assess their curriculum's effectiveness--knowing that these programs can prepare students to succeed in college-level writing is not the same as knowing that one's own program does so. OWCC's developmental writing course levels the playing field for its students, but the challenge remains to identify a more accurate placement method so that the college may better meet its students' academic needs.

Table 1

A Comparison Between Composition /Results of Group 1 (Developmental
Students) and Group 4 (Non-Developmental Students)

Measure of Effectiveness                         Group 1      Group 4
                                                 (n = 58)    (n = 792)

Pass rate for all students initially enrolled
  in the course                                   74%         63%
Pass rate for all students completing the
  course                                          84%         78%
Average grade for all students initially
  enrolled in the course                          74% (a)     68% (a)
Average grade for all students completing the
  course                                          77% (a)     79% (a)
Percentage of students withdrawing from the
  course                                          12%         18%
Percentage of completers earning an F (no
  credit)                                         14%         19%

(a) Note. Passing is a C, which is 75%. An A is 95%, a B is 85%, and an
F is 65% or lower.

No pluses or minuses are awarded.

Table 2

Grade Changes From College Prep English II to Composition I
for Group 1 (n = 58)

                f       f%

Same grade      17    29.3%
Lower grade     39    67.2%
Higher grade     2     3.4%

References

Berger, D. M. (1997). Mandatory assessment and placement: The view from an English department. In J. M. Ignash (Ed.), Implementing effective policies for remedial and developmental education (pp. 33-41). New Directions for Community Colleges, no. 100. Retrieved June 26, 2003, from WilsonSelectPlus database.

College Entrance Examination Board. (2002). The College Board: Accuplacer. Retrieved July 11, 2003, from http://cpts.accuplacer.com/docs/ CoordinatorGuide.html

Glau, G. R. (1996, March). Bringing them home: Arizona State University's new model of basic writing instruction. Paper presented at the annual meeting of the Conference on College Composition and Communication, Milwaukee, WI. (ERIC Document Reproduction Services No. ED403558)

Jenkins, A. & Boswell, K. (2002). State policies on community college remedial education: Findings from a national survey. Retrieved August. 8, 2003, from Education Commission of the States, Center for Community College Policy Web site: http://www.ecs.org/clearinghouse/40/81/4081.pdf

McCormick, F. & McCormick, C. (1986). The basic writing course at Eastern Illinois University: An evaluation of its effectiveness. WPA: Writing Program Administration, 10 (1/2), 61-65.

Office of Educational Services and Research, Division of Community Colleges. (1999). CCS college preparatory program. Retrieved March 6, 2003, from http://www.dcc.firn.edu/OSAS/Fast%20Facts/ff15.htm

Ragland, M. M. (1997). Leveling the playing field for developmental writers: A three year study. Research & Teaching in Developmental Education, 14(1), 5-11.

Schmitz, C. C., & delMas, R. C. (1991). Determining the validity of placement exams for developmental college curricula. Applied Measurement in Education, 4(1), 37-52. Retrieved June 26, 2003, from Academic Search Premier database.

Stein, J. B. (1982). Comparison of two methods of teaching fundamentals of written English and subsequent success in college composition courses at Minneapolis Community College. Minneapolis, MI: Minneapolis Community College. (ERIC Document Reproduction Service No. ED216720).

Weissman, J., Bulakowski, C., & Jumisko, M. K. (1997). Using research to evaluate developmental education programs and policies. In J. M. Ignash (Ed.), Implementing effective policies for remedial and developmental education (pp. 73-80). New Directions for Community Colleges, no. 100. Retrieved June 26, 2003, from WilsonSelectPlus database.

White, E. M. (1998). Teaching and assessing writing. Portland, ME: Calendar Islands.

Wolcott, W., & Legg, S. M. (1998). An overview of writing assessment: Theory, research, and practice. Urbana, IL: National Council of Teachers of English.

Anne Hay Southard is Director of Instructional Technology and Distance Learning at Okaloosa-Walton Community College and an instructor in the college's Communications/Social Science division. [e-mail]

Jennifer Clay is an instructor in Okaloosa-Walton Community College's Communications department and serves as the curriculum coordinator for the college's charter school, Collegiate High School. [e-mail]