A Perkins challenge: assessing technical skills in CTE.

[ILLUSTRATION OMITTED]

Current federal law requires states to develop performance measures and data collection systems for the four required core indicators, including technical skill attainment for secondary and postsecondary levels. These measures must be reliable and valid. Perkins IV requires states to use state-established, industry-validated career and technical skill standards; but few of these exist and states have largely been compelled to rely on career and technical education (CTE) course completion, grade point averages (GPA) in CTE courses, or administrative data.

Challenges

The first challenge posed by the Perkins requirement is defining a technical skill. How do we parse technical skills from non-technical skills? The most common approach relates technical skills to skill standards which are "performance objectives and competencies required by a specific occupation, as specified by experts within that particular industry" (Center for Remediation Design, 1991), and "performance specifications that identify the knowledge, skills and abilities an individual needs to succeed in the workplace." According to the National Skill Standards Board, skill standards "... consist of two components: (1) a description of the responsibilities needed for competent performance, and (2) a description of knowledge and skills necessary to carry out these responsibilities" (2000).

A second challenge is measurement. True performance assessments, sometimes known as authentic assessment, require a student to perform. Performing a task, the process of doing, rather than selecting an answer from a ready-made list (circle "D" for "none of the below") is a hallmark of performance assessments. The U.S. Congress Office of Technology Assessment defines performance assessment as "any form of testing that requires a student to create an answer or a product that demonstrates his or her knowledge or skills." This requires experienced raters who judge the quality of the student's performance against established criteria; consider driver certification tests or Olympic events. Closer to home, one need only look to the competitive events sponsored by CTSOs to see how this might be done. DECA events, for example, require both a test of factual knowledge and a performance.

An authentic performance assessment relies on experts among whom there is high inter-rater reliability. Those who have experience with CTSO competitive event judging know the inherent difficulty in this process. As well, a commentary in a recent issue of Education Week points out the manifold problems with scoring of open-ended questions on the National Assessment of Educational Progress that ought to give one pause before advocating such a system for CTE technical skill assessment. Another factor to consider in this process is logistics and attendant costs. Imagine if the SkillsUSA or DECA competitive events were adopted as the model to meet the Perkins accountability requirement. Also, there is the need to establish these measures as being both valid and reliable; this can be an extraordinarily expensive proposition.

Another challenge is timing. When is a student assessed? Presently, among states that do assess, they do so at the end of a program. The current evidence on high school student enrollments suggests that fewer students are completing programs than previously. Should assessments be done at the end of course to ensure we capture sufficient data to improve programs? Do end-of-course assessments in these high schools truly reflect technical skill attainment? Also, Perkins IV now requires that all states have programs of study/career pathways for all CTE students; connecting secondary and postsecondary CTE and assessing and reporting progress at the end of completing these pathways may be more relevant for determining progress and success in CTE.

An attendant issue is measurement to what end? That is, what use will be made of the data generated by the technical skill assessment? The National Research Center for Career and Technical Education convened a small panel three years ago to discuss this issue. The panel included industry representatives, experts in technical skill assessment, as well as leaders from the CTE community. Among the results of this discussion was a rank list of important assessment functions:

1. Provide data for local program improvement. (A diagnostic function.)

2. Provide a signal to the labor market or higher education. (A certification function.)

3. Provide data that can be aggregated within a state and among states. (An accountability function.)

Industry-Driven Models

Assessing technical skills is not a new requirement; over the past decades at least three models have emerged, with a fourth under serious consideration. Industry-driven models have long been a part of CTE. The auto industry, for instance, has developed a partnership with the National Automotive Technicians Education Foundation and a nonprofit organization Automotive Youth Educational Systems (AYES) that has established a set of curricular and programmatic standards designed to deliver high-quality, advanced skill training that is closely linked with workforce needs. Graduates of AYES-certified programs are eligible to be certified in brakes, steering and suspension, electrical systems and power train.

In the IT industry, industry leaders have created certification courses in networking and program language. Notable among these are Cisco, Novell, and A-Plus. Finally, the National Health Care Curriculum Consortium has partnered with the National Occupational Competency Testing Institute (NOCTI) to create a certification exam used in 35 states to assess knowledge critical to a family of health care occupations. The process requires the use of post-program assessments that provide either a certificate of proficiency or a certificate of mastery targeted at broad understanding of the health care industry and related content area. These are in addition to the widely recognized certified nursing assistant, licensed practical nurse or registered nurse certifications. These industry certifications provide a very clear signal to the labor market about a candidate's employability. Aside from industry produced assessments, third party organizations like NOCTI also provide industry-validated assessments.

State Models

Massachusetts is an example of a state model in which employability certification programs are developed in cooperation with employers. The state has identified nine competencies that form the foundation for its work-based learning plan. Successful attainment of these competencies meets a second kind of certification--employability. (The first certification is the academic Massachusetts Comprehensive Assessment System.) A third awaits those who successfully complete CTE programs in areas where the state has partnered with industry to develop industry-specific certifications.

In addition, states like Utah are developing end-of-course exams that presumably assess skills as well as knowledge. Success in applying these state models for measuring technical skill attainment can simultaneously meet the U.S. Department of Education's twin conditions of third-party assessment as well as validity and reliability. However, if the requirement to increase student and program coverage is also in place, state models to implement technical skill assessment structures can be cost prohibitive.

The CTSO Model

As noted earlier, each of the eight CTSOs include industry-validated competitive events in their portfolio of activities. Using cognitive tests of related content knowledge and performance activities that are industry judged and validated, the CTSO model presents another model for technical skill assessment. SkillsUSA is currently developing dozens of industry-based skill assessments. In general, where neither industry-driven models nor state models are feasible, states are increasingly turning to state-validated but locally applied assessment systems, which generally could be hybrids of the two models discussed above. Additionally, states have in place proxy measures for technical skill attainment such as the use of conventional academic success measures such as GPA, course completion and administrative data. While the latter methods of assessment are not commonly preferred under Perkins IV, many states do have them in place to measure technical skill attainment and have included them in their state Perkins plans arguing that these methods are as valid and reliable as the third-party technical skill assessments preferred under Perkins IV.

[FIGURE 1 OMITTED]

[FIGURE 2 OMITTED]

Challenges Ahead

There is a new effort that focuses on the 16 career clusters and the 81 career pathways. The challenges of definition and measurement are perhaps more pronounced at this level. While measuring a health cluster student's ability to "effectively communicate diagnostic information using written communication" may seem relatively straightforward, "employing occupational safety techniques" poses greater measurement challenges. Both are identified as technical skills in the current career clusters framework. One way to think about skill assessment is to envision a pyramid that combines a mix of academic knowledge and technical skills that changes as one moves from clusters to pathways to programs and on to specific occupational training (see figure 1). Using the health cluster as an example, one can envision a different kind of assessment for a cluster than one might create for a program (see figure 2). It is also likely that different constituencies (e.g., higher education, state government, teachers, industry and students) might attend to different assessments linked to different points on the pyramid. Many industries--for example health, auto or IT--might focus on program or occupation-specific education while education agencies might focus on clusters or pathways.

Given all of the above, basic and fundamental questions remain: what is the most important focus for high school and college CTE technical skill assessments? Should we focus on diagnostics, signaling or accountability? Is it possible to do all three with one assessment? These are important questions confronting the field as CTE struggles to identify the best approach that benefits our students, their (future) employers, and the several educational entities that have an interest in these questions.

References

Aragon, S. R., Woo, H.-J., & Marvel, M. R. (2004). Analysis of the Integration of Skill Standards into Community College Curriculum. St Paul, Minnesota. National Research Center for Career and Technical Education. (PDF 2,232KB)

Davis, J.L. (2006). Adopting Industry Skill Standards Can Strengthen CTE. Tech Directions, 66(3), 22-23.

Loyd, B., & Bailey, T. (2002). Transportation Skill Standards Project. Washington State Board for Community and Technical Colleges. Retrieved October 13, 2008, from www.wa-skills.com/pdfs/ transportation/Section01.pdf.

James R. Stone III is the director of the National Research Center for Career and Technical Education at the University of Louisville, Kentucky. He can be contacted at [e-mail].