Critical Academic Skills for Kansas Community College Graduates: A Delphi Study.

A three-round Delphi survey was used to gain consensus among 23 community college administrators and faculty representing 19 community colleges in Kansas. First-round participants defined 199 critical academic skills for community college associate's degree holders. Second- and third-round participants rated the skill statements based on a nine-point Likert scale and achieved consensus on seven skills that included key critical competencies defined by Gardner (1994). The authors suggest ways to use the Delphi technique to define outcomes for assessing institutional effectiveness.

Throughout the country, mandates that require community colleges to assess and improve their performance through reliable and valid effectiveness measures have increased and intensified. Legislators and the public are insistent that community colleges demonstrate accountability for tax dollars and that student outcomes should reflect public investments. Without significant input from the community colleges that they affect, however, policymakers can create external mandates counterproductive to the data community colleges use as critical indicators of effectiveness (Roueche & Roueche, 1998).

Traditional measures of institutional effectiveness such as resources and reputation have not fulfilled their intended purpose of demonstrating a return on what is invested in higher education (Cameron & Bilmoria, 1985; Ewell, 1994b, 1994c; Gardiner, 1994; Guskin, 1994; Lucas, 1996). Because the criteria used for evaluating research institutions often place two-year colleges at a disadvantage, community college leaders should be actively involved in establishing criteria for the effectiveness indicators upon which their institutions will be judged (Burke & Serban, 1997-98; Meyer, 1996; Roueche, Johnson & Roueche, 1997; Sorenson, 1996).

Based on a nationwide study, Roueche, Johnson, and Roueche (1997) concluded that pressure from powerful sources outside institutions compels the majority of community colleges to address institutional effectiveness questions related to what and how well students are learning with measures of quality that may differ from those used by baccalaureate institutions. According to Meyer (1996), an accountability perspective that considers student goals and outcomes such as workplace and literacy skills could be more fruitful than traditional measures of quality for evaluating and improving the performance of community colleges such as reputation and resources.

Scholars agree that although attainment of student goals and outcomes is an important consideration in determining institutional effectiveness, it is difficult to identify a discrete list of skills that should be the hallmark of a college graduate (Carpenter & Doig, 1988; Facione, 1990; Greenberg, 1988; Jones, 1994, 1995, 1996a; Jones & Melander, 1993; Jones & Ratcliffe, 1993; Jones & Tibbetts, 1993; Pascarella & Terenzini, 1991; Terenzini, 1997). Much of the literature addressing critical skills focuses on baccalaureate degree graduates rather than community college graduates. Few critical skills researchers acknowledge the developmental differences that exist between two-year and four-year college students; thus, stratified levels of skill attainment for learners across postsecondary environments are not usually specified (Facione, 1990; Jones, 1995; Jones & Melander, 1993; Jones & Tibbetts, 1993).

Critical academic skills are measures of student performance that specifically relate to intended cognitive outcomes of college graduates. Critical thinking, speech, writing, reading, group interaction, and quantitative reasoning skills have many overlapping components; thus, it is difficult to compile a list of discrete skills. Several researchers conclude that these overlapping elements further complicate measurement of skill attainment (Facione, 1990; Jones, 1995; Jones & Melander, 1993; Jones & Tibbetts, 1993).

Historically, faculty have been charged with assessing student performance. Therefore, it follows that faculty should provide leadership for developing a common set of critical academic skills that an institution should expect of its graduates (Dural, 1994; Ewell, 1994a; Hudgins, 1995, 1993; Roueche, 1997; Seybert, 1994; Sorenson, 1996). Faculty participation in establishing institutional assessment criteria was a requirement for the study described in this report, which was conducted in Kansas, a state where the impetus for community college effectiveness comes primarily from the legislature and the Kansas State Board of Education (KSBE). For more than two decades, the Kansas legislature struggled with the issue of who should supervise universities, community colleges, and other postsecondary institutions (Special Committee, 1997). Between 1972 and 1999, at least 22 reports presented recommendations for reforming the Kansas postsecondary education system (Efird & Rampey, 1997; Special Committee, 1997). Kansas legislators have acknowledged that although there have been enough studies, there has not been enough action to implement recommendations for reforming higher education in the state (Efird & Rampey, 1997).

In an effort to unify a fragmented system of public higher education, the Kansas Legislative Coordinating Council (LCC) authorized the Committee on Community College Governance to address governance and financial accountability issues for the state's community colleges (Mealer, 1997; Special Committee, 1997). In 1998, when the Committee on Community College Governance recommended to the legislature that the state increase its contributions to community college budgets from 27% to 39%, community colleges were charged to specify goals for the increased state funds (Special Committee, 1997). In an effort to meld community colleges into a statewide system, the Kansas legislature established accountability standards, staff and program development guidelines, and program definitions. To address concerns about cost effectiveness and program quality, the legislature asked each community college to develop a policy that defines institutional effectiveness and identifies indicators that confirm accountability.

Anticipating a legislative mandate for a common accountability measurement system, the Kansas Community College Council of Presidents and the Kansas Association of Community College Trustees initiated a process to identify 13 core indicators of effectiveness. A Task Force on Core Indicators consisting of 8 Kansas community college leaders, 2 legislators, and a Kansas State Board of Education representative delineated data elements and data sources for all community college effectiveness indicators except Kansas Core Indicator 4, Measurement of Critical Skills, the topic addressed by this study. This study was motivated by a need to define critical skills. The impetus for this specific study emerged from conversations among one of the authors and members of the Task Force on Core Indicators.

The study described in this report was designed to determine the critical academic skills that should be the hallmark of associate's degree graduates of Kansas community colleges. The research question that guided this study was as follows: What critical academic skills should be characteristic of Kansas community college graduates?

Methodology

The method of research chosen for this study was the Delphi technique. This method was selected because it can be used to gain consensus where geography limits the practicality of face-to-face discussions and when anonymity is desired.

All 19 community college deans at the 19 public community colleges in the state were contacted at a regularly scheduled meeting of the Kansas Community College Council of Instructional Administrators (KCIA) or by personal letters immediately following the meeting. These deans suggested the names for a panel of community college experts based on the following criteria: (a) more than three years teaching experience at the community college level; (b) a demonstrated commitment to teaching and learning through involvement in professional development activities and innovative classroom practices; (c) an instrumental role in assessment or institutional effectiveness programs or practices; and (d) opinions that are respected by peers and administrators and that provide a balanced voice for the institution.

Nineteen nominated faculty experts were contacted by personal letter. The letter invited them to participate in a three-round Delphi Study and identified the senior administrator who had nominated them. Also, the letter included an explanation of the study and provided an estimate of the time commitment for participation. Nominees were advised that each round of the study would require approximately 45 minutes and that data collection would occur over a two-month time period. Return of a signed consent form served as the panel member's agreement to participate in the study. Fifteen deans and 15 faculty members agreed to participate in the study.

In the first round of the Delphi Study, each panel member was asked for responses to the following question: "What are the critical academic skills which should be characteristic of Kansas community college associate degree holders?" Only those participants returning responses to the first-round question were contacted during the second round. For rating critical academic skills in the second and third rounds, each panel member was asked to use a nine-point Likert scale to rate the critical academic skills identified in the round with 1 designating not important and 9 designating very important. Panel members were directed to read the comments of other panel experts, to consider the degree of concurrence obtained for each item, and to defend their respective responses.

Likert scale responses were analyzed to determine the final list of critical academic skills. The median score was calculated for each of the critical academic skill statements. Consensus was obtained for a given critical academic skill when the median score of a skill statement equaled 9 and when 80% of respondents rated the critical academic skill in the upper third of the nine-point Likert scale (a rating of 7, 8, or 9).

There are a number of limitations to this study. This study is concerned with the identification of critical academic skills for Kansas community college graduates; thus, specific results are applicable only to the Kansas system. Another limitation is that the population from which the sample for this study was drawn reflects only two groups of stakeholders, community college faculty and chief instructional officers. A final limitation is that the opinions of faculty experts are not necessarily generalizable to all Kansas community college faculty members, nor are the opinions of participating deans of instruction generalizable to all chief instructional officers.

Findings

The Delphi research findings reported here resulted from the participation of 23 community college academic administrators and faculty (77% of the 30 experts who agreed to participate in the study). Fortythree percent of the responding panelists were deans or vice presidents of instruction from Kansas community colleges; 57% of the panelists were community college faculty considered to be knowledgeable about curriculum, instruction, and assessment. Major teaching assignments for faculty panelists were English (45%); business (18%); sciences (18%) and fine arts (9%). The remaining 18% of faculty participants were assessment directors with either past or concurrent assignments as faculty. Return rates for the three rounds of the survey and cumulative percent of return are presented in Table 1.

Table 1 Critical Academic Skills Study Response Rates by Round

                          No. of      Percentage of
Delphi   No. of Mailed    Returned    Questionnaires   Cumulative
Round    Questionnaires   Responses   Returned         Percentage

  1            30            23             77             77
  2            23            21             91             70
  3            21            20             95             67

As the data in Table 1 indicate, some panel members dropped out; thus, Rounds Two and Three had fewer respondents than Round One. Although the total response rate of 67% is lower than would be acceptable in a conventional survey, compared to other Delphi studies that report final response rates of less than 50%, this is considered an acceptable level of response. Participants were treated as one group for data analysis and interpretation purposes.

During the first round, 23 respondents identified 199 critical academic skills. To provide a framework for analyzing the critical academic skills generated in Round One, the investigators developed 13 categories (Table 2) based on similarities among panelists' responses.

Table 2 Critical Academic Skill Categories Identified in Round One of a Three-Round Delphi Study

Critical Academic Skill Categories      No. of Skill Statements
                                              by Category

Mathematics                                        42
Problem-Solving and Critical Thinking              39
Writing                                            16
Culture and Society                                15
General                                            13
Communication (general)                            11
Oral Communication                                 11
Group Interaction                                  11
Citizenship                                         9
Reading                                             9
Technology                                          8
Physical Well-Being                                 7
Miscellaneous                                       3

Total                                             199

As shown in Table 1, 21 (91%) of the 23 first-round participants returned responses in Round Two. The 199 critical academic skill statements generated from the first round of responses were reduced to 16 critical academic skills statements during Round Two. Panelists' comments related to these 16 critical academic skills statements were written below each item so that respondents would consider the remarks of other panel members as they made final ratings in Round Three. At no point during the Delphi process were comments edited or statements synthesized.

After three rounds, 20 panelists (87% of the original 23 panelists) achieved consensus on seven critical academic skills, six of which were unduplicated. Based on the comments of several respondents, two of the seven skills were combined in the final list of skills. The critical academic skills identified by the panel were as follows:

   Kansas community college associate degree holders should be able to
   demonstrate the ability to clearly communicate thoughts, complex ideas, and
   questions both orally and in writing; an awareness of cultural and social
   diversity and how cultural differences impact and influence assumptions,
   perceptions, and personal values; the ability to think critically and make
   reasonable judgements by acquiring, analyzing, synthesizing, and evaluating
   information; the ability to add, subtract, multiply, and divide whole
   numbers, fractions, decimals, and percentages; and the ability to work
   cooperatively with others and effectively as an individual.

Discussion

The finite list of critical academic skills for Kansas community college graduates produced through consensus of an expert panel included key elements of "critical competencies" for baccalaureate degree graduates identified by Gardiner (1994). Gardiner's critical competencies include oral and written communication skills, interpersonal and team skills, critical thinking and complex problem-solving skills, and respect for people different from oneself.

The skills in Gardiner's list for college graduates that were not included in the list generated by this three-round Delphi study are dispositional skills. The specific dispositional skills in Gardiner's critical competencies for baccalaureate degree holders are (a) conscientiousness, personal responsibility, and dependability; (b) the ability to act in a principled, ethical fashion; (c) the ability to adapt to change; and (d) the ability and desire for lifelong learning. Although many panelists in this study wanted to include dispositional skills such as the ability to act in a principled and ethical fashion, there was general agreement that these skills did not constitute academic skills. They perceived the skills to be critical; however, they concluded that dispositional skills did not fit the definition of "critical academic skills" provided to the panel of experts.

Moreover, a debate arose among the Round Two panelists over the inclusion of technical skills in the list. Jones (1996b) addressed the concept of technical skills, suggesting that "although employees enter new positions with adequate technical skills, it is the generic skills (especially communication and problem-solving) that count toward successful job performance over time, and these skills are most often absent" (p. 8).

Four concerns were voiced by the panel of experts in all rounds: (a) level of skill attainment; (b) measurability; (c) redundancy; and (d) degree of manageability of the finite list of critical academic skills.

Level of Skill Attainment

The level of critical academic skill attainment for community college graduates remains an important concern for educators. The findings in this study reflect discussions by scholars concerning developmental differences both among community college students and between two-year and four-year graduates. In a national Delphi study conducted by the National Center on Postsecondary Teaching, Learning and Assessment at Pennsylvania State University, Jones (1995, 1996a) concluded that although differences do exist between upper and lower division students, outcomes to reflect this dichotomy are not usually specified. Rubin and Morreale (1996) assert that there are common expectations that skill levels are developed and refined throughout one's college experiences, an opinion voiced by Kansas community college experts in this study. Palmer (1993) and Spicer and Hanks (1995) contend that a primary measurement concern is finding a balance between too many levels of skill attainment and a single skill attainment level that is set either unrealistically high or low.

Measurability

The issue of measurability of critical academic skills was a source of frustration for several participants. This study was not designed to determine data sources necessary for measuring the identified critical academic skills. Participants were asked; however, to consider and report critical academic skills that they perceived could be documented through data sources used in a statewide common data collection and reporting system. As Terenzini (1997) found, it is of far greater value to define what will be measured before tackling the stickier methodological questions of how and how easily information on a critical academic skill can be gathered. Ewell and Jones (1991) assert that developing direct assessments of a college graduate's ability to "think critically, communicate effectively, and solve problems" is technically complex and takes a minimum of five years from initiation to implementation.

Redundancy

Several respondents expressed frustration concerning repetitive skill statements. This is a limitation of the conventional Delphi methodology used. Not sanitizing statements or collating responses increases the reliability of the study; this practice also requires a high level of discernment on each panelist's part.

Manageability

The desire for a manageable list of skills was expressed by several panelists. One respondent commented, "The list of critical skills should be manageable (in other words, 5 to 10)." The practice of limiting the number of outcomes to be assessed to those that are essential and most meaningful also is recommended by Pratt, Reichard, and Rogers (1989).

Recommendations

This research report represents efforts in one state toward defining outcomes that can be measured to assess institutional effectiveness and thus meet public and legislative demands for accountability. Based on the findings in this study, the authors offer the following recommendations:

1. The general studies curriculum for community college students should be carefully examined using the list of critical academic skills and considering multiple goals of associate's degree students. Results of a general studies curriculum analysis may suggest that individual courses in critical thinking, problem-solving, and communication are warranted; however, infusing the identified critical academic skills into existing courses is a practical alternative.

2. Community college faculty should be encouraged to modify courses to address both directly and indirectly each critical academic skill area and eventually to develop annotated syllabi and delineate specific course objectives that reflect the identified critical skills.

3. Community college faculty should be encouraged and supported in their efforts to assess each of these critical academic skills within the context of individual courses and programs.

Summary

This study documents an important time in the history of Kansas community colleges in that institutional representatives came together to discover if some degree of general accord could be found in defining common academic outcomes for state community college graduates. The purpose of the study was to define a finite list of critical academic skills that should be the hallmark of Kansas community college graduates. A conventional Delphi technique using three rounds was the procedure chosen for data collection and analysis. Although panel members expressed frustration in attempting to define a finite list of critical academic skills for community college graduates, they recognized the need to specify such outcomes unique to community college graduates with special attention to the multiplicity of goals of this population. After three rounds, the community college panel of experts achieved consensus on five unduplicated critical academic skills.

The information gathered in this research should be useful to Kansas community college policymakers including faculty and administrators as they seek to determine assessment methods for the critical academic skills identified in this study. Using a modified Delphi technique with a range of standardized responses could help focus attention on specific identified critical academic skills areas and allow participants to address the stickier methodological issues of measurement. The research may be of assistance to institutional policymakers in other states and institutions as they attempt to construct common data measurement systems within and across states.

References

Burke, J., & Serban, A. (1997-98, December/January). Performance funding: Fashionable fad or emerging trend? Community College Journal, 68(3), pp. 26-29.

Cameron, K., & Bilmoria, D. (1985). Assessing effectiveness in higher education. Review of Higher Education, 9(1), 101-118.

Carpenter, C., & Doig, J. (1988). Assessing critical thinking. In J.H. McMillan (Ed.), Assessing student's learning (pp. 33-46). New Directions for Teaching and Learning, No. 34. San Francisco: Jossey-Bass.

Duval, B. (1994). Obtaining student cooperation for assessment. In T. Bers and M. Mittler (Eds.), Assessment and testing: Myths and realities (pp. 47-52). New Directions for Community Colleges, No. 88. San Francisco: Jossey-Bass.

Efird, J., & Rampey, C. (1997. May 27). Past studies and recommendations to reorganize postsecondary education in Kansas. Topeka, KS: Kansas Legislative Research Department.

Ewell, P. (1994a). Developing statewide performance indicators for higher education: Policy themes and variations. In S.S. Ruppert (Ed.), Charting higher education accountability: A sourcebook on state-level performance indicators (pp. 147-166). Denver, CO: The Education Commission of the States. (ERIC Document Reproduction Service No. ED 375 789)

Ewell, P. (1994b, November/December). A matter of integrity: Accountability and the future of self-regulation. Change, 26, pp. 25-29.

Ewell, P. (1994c). The assessment movement: Implications of teaching and learning. In T. O'Banion (Ed.), Teaching and learning in the community college (pp. 73-96). Washington, DC: American Council on Education.

Ewell, P., & Jones, D. (1991). Actions matter: The case for indirect measures in assessing higher education's progress on the National Education Goals. Boulder, CO: National Center for Higher Education Management Systems. (ERIC Document Reproduction Service No. ED 340 756)

Facione, P. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. (ERIC Document Reproduction Service No. 315 423).

Gardiner, L. (1994). Redesigning higher education: Producing dramatic gains in student learning. ASHE-ERIC Higher Education Report, Volume 23, No. 7. Washington, DC: The George Washington University, Graduate School of Education and Human Development.

Greenberg, K. (1988). Assessing writing: Theory in practice. In J. H. McMillan (Ed.), Assessing students' learning (pp. 47-66). New Directions for Teaching and Learning, No. 34. San Francisco: Jossey-Bass.

Guskin, A. (1994, July/August). Reducing student costs and enhancing student learning: The university challenge of the 1990s. Change, 26, pp. 23-29.

Hudgins, J. (1995, October 30). Using indicators of effectiveness to demonstrate accountability of community colleges. Paper presented at a meeting of the Texas Association of Community College Trustees and Administrators. Austin, TX. (ERIC Document Reproduction Service No. ED 394 602)

Hudgins, J. (1993, April/May). Institutional effectiveness: A strategy for renewal. AA CC Journal, pp. 41-44.

Jones, E. (1996a, November/December). Communications outcomes expected by faculty, employers, and policymakers. Assessment Update, 8(6), 7-8.

Jones, E. (Ed.). (1996b). Preparing competent college graduates: Setting new and higher expectations for student learning. In E. Jones (Ed.), National and state policies affecting learning expectations (pp. 7-18). New Directions in Higher Education, No. 96. San Francisco: Jossey Bass.

Jones, E., & Associates. (1995). National assessment of college student learning: Identifying college graduates' essential skills in writing, speech and listening, and critical thinking. Washington, DC: United States Department of Education and Office of Educational Research and Improvement, NCES Document 95-001. (ERIC Document Reproduction Service No. ED 383 255)

Jones, E. (1994). Defining essential writing skills for college graduates. Innovative Higher Education, 19(1), 67-77.

Jones, E., & Melander, L. (1993). Speech communication skills for college students. University Park, PA: National Center on Postsecondary Teaching, Learning and Assessment, Pennsylvania State University. (ERIC Document Reproduction Service No. ED 358 773)

Jones, E., & Ratcliffe, J. (1993). Critical thinking skills for college students. University Park, PA: National Center on Postsecondary Teaching, Learning and Assessment, Pennsylvania State University. (ERIC Document Reproduction Service No. ED 358 774)

Jones, E., & Tibbetts, S., 1993. Writing skills for college students. University Park, PA: National Center on Postsecondary Teaching, Learning and Assessment, Pennsylvania State University. (ERIC Document Reproduction Service No. ED 358 775)

Lucas, C. (1996). Crisis in the academy. New York: St. Martin's Press.

Mealer, B. (1997, November 14). Kansas may change governance of community colleges. The Chronicle of Higher Education, 44(12), pp. A34.

Meyer, M. (1996, June 25). Institutional effectiveness analysis and student goal attainment in the community college. Paper presented at the Consortium for Community College Development, Charleston, SC. (ERIC Document Reproduction Service No. ED 395 643)

Palmer, J. (1993). Institutional accreditation, student outcomes assessment, and the open-ended institution. In C. Prager (Ed.), Accreditation of the two-year college (pp. 49-59). New Directions for Community Colleges, No. 83. San Francisco: Jossey-Bass.

Pascarella, E., & Terenzini, P. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass.

Pratt, L., Reichard, D., & Rogers, B. (1989). Setting intended educational instructional, research, and service outcomes and administrative objectives. In J.O. Nichols (Ed.), Institutional effectiveness and student outcomes assessment implementation on campus: A practitioner's handbook (pp. 102-119). New York: Agathon Press.

Roueche, J. (1997, October). Leadership challenges for 2000. Paper presented at the meeting of the Kansas Association of Community College Trustees, Wichita, KS.

Roueche, J., Johnson, L., & Roueche, S. (1997). Embracing the tiger: The effectiveness debate in the community college. Washington, DC: American Association of Community Colleges.

Roueche, J., & Roueche, S. (1998, April/May). Dancing as fast as they can. Community College Journal, 66(5), 30-35.

Rubin, R., & Morreale, S. (1996). Setting expectations for speech communication and listening. In E. Jones (Ed.), Preparing competent college graduates: Setting new and higher expectations for student learning (pp. 19-30). New Directions in Higher Education, No. 96. San Francisco: Jossey-Bass.

Seybert, J. (1994). Assessment from a national perspective: Where are we, really? In T. Bers and M. Mittler (Eds.), Assessment and testing: Myths and realities (pp. 23-30). New Directions for Community Colleges, no. 88. San Francisco: Jossey-Bass.

Sorenson, R. (1996, February 14-17). What are the roles of the chair, faculty member and dean in student assessment? In the Olympics of Leadership: Overcoming obstacles, balancing skills, taking risks. Proceedings of the Annual International Conference of the National Community College Chair Academy. Phoenix, AZ. (ERIC Document Reproduction Service No. ED 394 599)

Special Committee on Community College Governance (1997). Structure and funding of postsecondary education, Final Report. Topeka, KS: Kansas Legislative Research Department.

Spicer, K., & Hanks, W. (1995). Multiple measures of critical thinking skills and predisposition in assessment of critical thinking. (ERIC Document Reproduction Service No. ED 391 185)

Terenzini, P. (1997). Student outcomes information for policy-making. Final report of the National Postsecondary Education Cooperative Working Group on Student Outcomes for a Policy Perspective. NCES Document 97-991. Washington, DC: National Postsecondary Education Cooperative; National Center for Education Statistics. (ERIC Document Reproduction Service No. ED 411 605)5.

Elizabeth Larson serves as academic dean of Zarem/Golde ORT Technical Institute in Chicago, Illinois. [e-mail]

Janice R. Wissman is a professor and associate dean of the College of Education at Kansas State University, Manhattan, Kansas. [e-mail]