Rehabilitation counseling accreditation: validity and reliability.

Measurement is "a means in the process of description, differentiation, explanation, prediction, diagnosis, decision making, and the like" (Pedhazur & Schmelkin, 1991, p. 15). Accreditation is, thus, a form of measurement and must be evaluated accordingly (Linkowski & Szymanski, 1993). The accreditation process applied to rehabilitation counselor education programs differs markedly from the processes used by other counseling disciplines. Specifically, it contains procedures to address validity and reliability of measurement. The purpose of this article is to explain rehabilitation counseling accreditation from the perspective of measurement. To this end, the following topics are addressed: (a) accreditation functions and issues, (b) validity of rehabilitation counselor accreditation, (c) reliability of the 1992-1993 Council on Rehabilitation Education process, and (d) implications.

Accreditation Functions and Issues

Accreditation is one way in which professions mark their boundaries. The educational standards, which are a part of accreditation, also theoretically benefit society through the preparation of professionals with more specialized knowledge (Rothman, 1987). However, there is increasing concern that the very prescription of educational standards that characterizes accreditation may actually impede the solution of social problems by the professions (see e.g., Armour & Fuhrmann, 1993; McGuire, 1993; Skrtic, 1991).

Essentially, accreditation is the satisfaction by professional education programs of minimum requirements in such areas as curriculum, faculty credentials, financial stability, admission and records, student evaluation, and administrative practices. "Accreditation requirements are imposed by peers--institutions and persons. They are valid by consensus and zeitgeist" (McGaghie, 1993, pp. 235-236).

The agents of accreditation are accrediting bodies. They regulate educational institutions (e.g., universities,

technical schools) and specialized training programs (e.g., counseling, dentistry) and are, themselves, regulated. The Council on Post Secondary Accreditation (COPA) had served to regulate a wide range of institutional and professional accrediting bodies (COPA, 1990). It was formed as an umbrella organization in 1975 to bring together several accrediting interests, including regional, institutional, and specialized program accreditation. Although this confederation lasted 18 years, considerable tension existed among these various levels of interests. And, as a result, in March 1993, the COPA Board voted to dissolve (Magner, 1993).

Successors to COPA have emerged (Atwell, 1994; Rees, 1993), and accreditation remains a topic of considerable national interest (Longanecker, 1994). The 1992 Amendments to the Higher Education Act directly affected accreditating agencies and echoed an "era of greater public accountability" (Longanecker, 1994, p. 13). Assessment of student outcomes is now a strong consideration on campuses, spurred in part by requirements of the regional organizations that accredit colleges and universities (Ewell, 1994).

Assessment of student outcomes is a key component in the accreditation process of the Council on Rehabilitation Education (CORE), which employs reliability and validity checks and uses multi-stakeholder input. The CORE process may be an exemplar for other accrediting organizations. This process and its application to the 1992-1993 accreditation cycle are described in the next sections.

Validity of Rehabilitation Counseling Accreditation

Evidence of validity of the CORE curricular standards is contained in a joint special issue of the Rehabilitation Counseling Bulletin and the Journal of Applied Rehabilitation Counseling (Szymanski & Leahy, 1993). As explained in the special issue, the Foundation for Rehabilitation Certification, Education, and Research has funded an ongoing joint research effort of CORE and the Commission for Rehabilitation Counselor Certification (CRCC). The results of the joint effort provide data to recommend new standards and validate current standards. A brief overview of the rationale, design, and initial results of the project is provided. Readers are referred to the special issue for more specific details.

Validity refers to the relative applicability of the inferences made on the basis of an instrument (Pedhazur & Schmelkin, 1991). Thus, instruments vary in validity across groups of people (Anastasi, 1988) and over time as the construct of interest evolves. This variation over time is an important caveat for accreditation. Changes over time have been demonstrated in the rehabilitation counseling profession (see e.g., Leahy, Szymanski, & Linkowski, 1993; Muthard & Salomone, 1969; Rubin et al., 1984). These changes are natural evidence of the evolution of the profession (Abbott, 1988; Rothman, 1987). Nonetheless, the instruments used in accreditation of rehabilitation counseling must be sensitive to such temporal variation. In other words, an instrument developed and validated this year cannot be presumed to be valid next year.

The research process developed by the joint CORE/CRCC project has provided a mechanism to routinely update and validate an instrument reflecting the knowledge standards underlying both CORE accreditation and CRCC certification (Szymanski, Linkowski, Leahy, Diamond, & Thoreson, 1993). The instrument combines a fixed response format with analysis of answers to open-ended questions. An expert panel (DeVellis, 1991) of CORE and CRCC commissioners was used to determine the degree to which the basic 55 instrument items represented the CORE and CRCC knowledge standards. Analysis of the open-ended responses has been used to determine new items to be tested for their importance to rehabilitation counseling practice. This is an ongoing process with new items being regularly developed and evaluated (Linkowski et al., 1993). Those that demonstrate sustained importance across work settings and over time will be considered by the CORE and CRCC commissions for inclusion as curricular standards.

The research instrument is administered every six months to rehabilitation counselors applying for their five-year renewal of certification. It is also administered to applicants for certification and to recent graduates of CORE accredited programs that are being evaluated for continued recognition. Regular data collection from these three sampling frames (maintenance, applicants, CORE graduates) has been institutionalized into the CRCC and CORE procedures. Data are collected from over 1,000 participants per year.

A combination of research designs is used in the CORE/CRCC

project. The research design for the ongoing study combines

descriptive, ex post facto, and time-series approaches. The

descriptive portion involves (a) examination of the perceived

importance of each of the items for each sampling frame at each

sampling interval, and (b) the use of data reduction strategies to

examine the factor structure underlying the knowledge areas.

The ex post facto nature of the study involves comparison of

responses to items and factor scores across a variety of

dimensions including sampling frames, levels of educational

preparation, employment setting, and length of experience in

rehabilitation counseling. The time-series portion of the design,

which will not begin until three years into the study, will

involve annual examination of patterns of change in the items

over time. . . . The combination of these designs will allow an

annual snapshot of rehabilitation counseling knowledge to be

recorded and an ongoing analysis of changes in importance of

knowledge areas. (Szymanski, Linkowski, Leahy, Diamond, &

Thoreson, 1993, pp. 115-116)

Initial results of the research were reported in the joint special issue (Szymanski & Leahy, 1993). The results provide evidence for a central core of rehabilitation counseling knowledge with some outlying areas that vary in importance across job settings (Leahy, Szymanski, & Linkowski, 1993; Szymanski, Linkowski, Leahy, Diamond, Thoreson, & Witty, 1993). Additionally, the efficacy of graduate education was supported by the fact that CORE program graduates reported higher levels of preparedness in the following 6 of 9 core components of rehabilitation counseling knowledge than did counselors with unrelated master's or bachelor's degrees: individual counseling and development, group and family counseling, foundations of rehabilitation, medical and psychosocial aspects of disability, workers' compensation, employer services and technology (Szymanski, Leahy, & Linkowski, 1993). These results, when combined with the procedures used for instrument development, provide evidence for the validity of the knowledge standards used in the CORE accreditation process and the CRCC examination content.

The most important advance of the CORE process is that regular data collection has been built into the CORE and CRCC procedures. This ongoing data collection provides input from persons in a variety of rehabilitation counseling job settings and job titles to measure changes in importance of specific knowledge areas across settings and titles and over time. Annual analysis of the obtained data is used to provide evidence of continued validity of the CORE curriculum standards. In addition, analysis of trends for each item, including new items, over time and across settings provides input for regular updating of accreditation and certification standards.

Reliability of the 1992-1993 Council on Rehabilitation Education Process

In addition to validity of its standards, the research committee of CORE has developed procedures to assure reliability of both its measures and its process. A measure cannot be valid unless it is reliable for the intended use. Reliability refers to the degree to which scores are free from errors of measurement (Pedhazur & Schmelkin, 1991). Two types of reliability are measured in the regular evaluation of the CORE process -- the internal consistency reliability of the various subscales of the instruments and the interrater agreement of the ratings of the review team members. The different methodologies are described in the following section.

Internal Consistency Reliability

Method. Measurement involves the attachment of numbers to aspects of constructs (Pedhazur & Schmelkin, 1991). For CORE, our construct of interest is rehabilitation counselor education; the aspects that are measured are the facets or standards of the process as operationalized in accreditation. Each instrument or version of an instrument reflects the views of a stakeholder group involved in the CORE process (i.e., graduates, students, internship supervisors, employers, faculty) regarding educational processes or outcomes. Similarly, each subscale corresponds with a specific CORE standard or substandard.

Internal consistency reliability, as measured by Cronbach's alpha, refers to the degree of relative homogeneity of the items in the subscale (Pedhazur & Schmelkin, 1991) or the proportion of total item variance due to actual variation across individuals (DeVellis, 1991). In essence it measures the degree to which the subscale items hang together. Such internal consistency is a prerequisite for valid measurement. Values for Cronbach's alpha range from 0 to 1. Values between .70 and .80 can be considered respectable, and values between .80 and .90 are very good (DeVellis, 1991).

The CORE process uses two different instruments. The multistakeholder instrument contains 101 items that represent the following groups of CORE standards: mission and objectives, general curriculum content, clinical experience, educational outcomes, and administration and faculty. Two response formats are used depending on the nature of the item. Items reflecting program characteristics have three possible response formats: yes, no, not sure. Items reflecting adequacy of preparation have five possible responses: no preparation, little preparation, moderate preparation, high degree of preparation, and very high degree of preparation. The stakeholder groups completed different items according to the appropriateness of the item to the stakeholder's frame of reference.

The heterogeneity of the stakeholder groups suggested separate reliability analyses for each of the five forms of the instrument. Subscales were determined according to the CORE substandards, which the instrument was created to measure.

The second instrument used in the CORE process was the CORE/CRCC research instrument (Linkowski et al., 1993). Although subscales of that instrument were empirically derived in the CORE/CRCC research study, a different item grouping was used in the evaluation of the reliability of the CORE process. Specifically, it was necessary to use the specific groupings of items by CORE standards and substandards, because that grouping is used by the review teams for evaluation of standards compliance. Lower alpha coefficients in this instrument or in the multi-stakeholder instrument may suggest regrouping of substandards.

Results and discussion. The internal consistency (Cronbach's alpha) indices of both the multi-stakeholder instrument and the CORE/CRCC research instrument are presented in Table 1. The multi-stakeholder instrument is reflected in the top portion of the table.

Table 1 Internal Consistency Reliability (Cronbach's Alpha) of Scales of Multi-Stakeholder Questionnaire

                                                Stakeholder Group

Scale/Standards                Items(a)         Graduate   Student

Program Provided                 1-9              .78       .74
  (A.1, C.1, C.3, C.4)

Clinical Experience              10-15            .68       .80
  (D.1.1-D.1.3)

Internship Experience            16-15            .75        NA
  (D.1.4-D.2.1)

Supervisor Perception            1,13,15,16,       NA        NA
  (A and D)                      17,19,20,22

Educational Outcomes
  E.1 Legal, Ethical             23-27            .86        NA
      History
  E.2 Counseling                 28-35            .91        NA
  E.3 Planning/                  36-51            .95        NA
      Case Management
  E.4 Voc/Career                 52-61            .93        NA
  E.5 Assessment                 62-76            .96        NA
  E.6 Placement                  77-87            .95        NA
  E.7 Research                   88-91            .92        NA

Program Provided                 92-101           .79        .84
   (F)
Graduates Prepared
 with Knowledge of:
   C.2.1 Foundations             G1-9             .89
   C.2.2 Counseling Services     G 10-23          .92
   C.2.3 Case Management         G24-28           .81
   C.2.4 Vocational and Career   G29,31           .70
         Development
   C.2.5 Assessment              G30, G31-37      .92
   C.2.6 Job Development and     G42,43, G45-47   .94
         Placement               G49-52
   C.2.7 Research                G53-55           .87

Scale/Standards                Supervisor   Employer   Faculty

Program Provided
  (A.1, C.1, C.3, C.4)             NA          NA        .76

Clinical Experience                NA          NA        .89
  (D.1.1-D.1.3)

Internship Experience              NA          NA         .90
  (D.1.4-D.2.1)

Supervisor Perception             .25          NA         NA
  (A and D)

Educational Outcomes
  E.1 Legal, Ethical              .81         .83         NA
      History
  E.2 Counseling                  .88         .91         NA
  E.3 Planning/                   .92         .93         NA
      Case Management
  E.4 Voc/Career                  .91         .93         NA
  E.5 Assessment                  .93         .95         NA
  E.6 Placement                   .93         .94         NA
  E.7 Research                    .91         .94         NA

Program Provided                  .84          NA         NA
   (F)
Graduates Prepared
 with Knowledge of:
   C.2.1 Foundations
   C.2.2 Counseling Services
   C.2.3 Case Management
   C.2.4 Vocational and Career
         Development
   C.2.5 Assessment
   C.2.6 Job Development and
         Placement
   C.2.7 Research

(a) Items refer to the numerical order of items on the master quesitonnaire. Items with a "G" are from the CORE/CRCC Research Questionnaire.

Results for the multi-stakeholder instrument were impressive. All but two Cronbach's alpha values were in the respectable range (i.e., .70 or above; DeVellis, 1991). Values of the educational outcomes subscales (E.1 through E.7) were quite high-across respondent groups, indicating excellent reliability of those subscales. Most other subscales demonstrated at least respectable reliability across groups. A major exception, however, was the clinical experience subscale. It was minimally acceptable for graduates, although it was very good for students and faculty. This variation suggests some attention to wording of the items to be sure that they mean the same thing to each stakeholder group. In addition, extremely low reliability was demonstrated by the composite scale, supervisor perception, which was developed to include those items in the first three subscales that were considered appropriate for rating by internship supervisors. The alpha value of .25 indicates that the scale lacks internal consistency and should be reconsidered.

Results for the CORE/CRCC research instrument were also impressive. Internal consistency reliability coefficients were all in the respectable range. The relatively low (.70) coefficient for the two-item scale, C2.4 suggests attention. Some of the items that are included in the evaluation of standard C2.5 are vocational in nature and would seem to belong better with standard C2.4 Minor regrouping of the items comprising these two standards should be explored.

Interrater Agreement

Method. The CORE process involves assignment of two-to three-person review teams to each school. Each member of the review team receives a self study document prepared by the school and results of the multi-stakeholder questionnaire. Based on this information, reviewers determine a program's degree of compliance with each of CORE's 36 standards according to the following scale: meets standard, needs improvement, does not meet standard, and insufficient evidence. The review team chair completes an individual review, and then, after reviewing team members' responses completes a composite review. The chair's composite review reflects additional consideration of the material, discussion of areas of disagreement with team members, and incorporation of additional information.

Additional important evidence of the reliability of the CORE process is obtained from examining the interrater agreement of the two to three independent reviewers comprising each review team. Low interrater agreement can indicate poor training of reviewers, poor definition of the subject of the observation (e.g., the CORE standard), problematic definition of the rating scale, or a host of other problems (Borg & Gall, 1989; Pedhazur & Schmelkin, 1991). Essentially, measures of interrater agreement allow CORE to regularly evaluate the reliability of the rating process and its components.

Interrater agreement was computed by a simple percent agreement formula (Borg & Gall, 1989), i.e., the number of agreements divided by the number of observations. The percent agreement for each school was the average of the three pair-wise interrater agreement percentages computed over the 36 standards. These school interrater agreement indices provide direct evidence of the relative reliability of the review teams.

Interrater agreement was also computed for each of the seven major groups of standards. This figure was computed from the number of agreements within review teams divided by the number of comparisons. The resultant index provides evidence of the relative reliability of ratings across standards. Low percent agreements may indicate an unclear standard.

Results and discussion. The results of the interrater agreement computations were not quite as stellar as the internal consistency reliability. The average interrater agreement for the review teams was 57%. When inferences based on observation are required, 70% is considered satisfactory (Borg & Gall, 1989). The safeguards in the CORE process, specifically the resolution of disagreements by review team chairs and review by a second review team at the annual meeting, may lessen the effects of such low reliability. Nonetheless, the relatively low interrater agreement suggests potential improvements to the CORE process.

As indicated in Table 2, two standards appear to warrant immediate attention. The low percent agreement for the B Standards, Program Evaluation (37.3%), and the D Standards, Clinical Experience (39.8%), indicate room for improvement. Attention should be directed to (a) the wording for the standards and (b) the directions to the schools for uniform presentation of evidence regarding the standards.

Table 2
Average Percent Agreement for Major Standards

General Standard                        Percent Agreement

A.      Mission and Objectives                76.6
B.      Program Evaluation                    37.3
C.      Curriculum                            68.4
D.      Clinical Experience                   39.8
E.      Educational Outcomes                  55.7
F.      Administration and Faculty            59.4
G.      Program Support and Resources         50.3

Overall Average                               57.0

Interrater agreement of the review teams also varied as demonstrated in Table 3. Although three review teams demonstrated greater than 70% agreement, two teams demonstrated less than 40%. The CORE safeguards lessen the effect of this lower reliability on the decision process. Nonetheless, there is clear room for improvement. Attention should be directed to training of raters and to clarification of the rating scale. Specifically, a process must be put into place to train new reviewers to an established criterion of performance (e.g., 80% agreement with an established review team). In addition, bench marks must be developed to clarify the application of each of the four rating categories (i.e., meets standard, needs improvement, does not meet standard, insufficient evidence) to each of the 36 standards.

Table 3
Average Percent Agreement by School

School      Percent Agreement

  1.              48.0
  2.              58.3
  3.              63.0
  4.              64.6
  5.              30.3
  6.              54.3
  7.              71.7
  8.              61.0
  9.              78.0
 10.              37.3
 11.              45.3
 12.              76.0
 13.              65.7
 14.              65.3
 15.              52.7

Overall Average   57.0

Implications

Evidence exists to support the validity of the CORE standards and the reliability of the CORE process. Improvements in the CORE process are recommended to increase reliability. Nonetheless, it should be noted that the empirical evidence presented in this portion of the paper is far more extensive than that used to document many other accreditation processes.

The research model upon which CORE developed its accreditation process and standards more than twenty years ago continues to be utilized to improve and adapt the evaluation process and to update standards for the 1990s and beyond. This systematic empirical approach and continuous involvement of multiple stakeholders can be a model for other professional fields and for higher education in demonstrating responsiveness to both accountability and change.

References

Abbott, A. (1988). The system of professions: An essay on the division of expert labor. Chicago, L: University of Chicago Press.

Anastasi. A. (1988). Psychological testing (6th ed.). New York, NY: Macmillan.

Armour, R. A., & Fuhrmann, B. S. (1993). Confirming the centrality of liberal learning. In L. Curry, J. F. Wergin, & Associates, Educating professionals: Responding to new expectations for competence and accountability (pp. 126-147). San Francisco: Jossey-Bass.

Atwell, R. H. (1994). Putting our house in order. Academe, 80(4), 9-12.

Borg, W. R., & Gall, M. D. (1989). Educational research (5th ed.). New York, NY: Longman.

Council on Postsecondary Accreditation (COPA). (1990). The COPA handbook. Washington, DC: Author.

Curry, L., & Wergin, J. F. (1993). Setting priorities for change in professional education. In L. Curry, J.F. Wergin, & Associates, Educating professionals: Responding to new expectations for competence and accountability (pp. 316-327). San Francisco: Jossey-Bass.

DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park, CA: Sage.

Ewell, P. T. (1994). Regaining our balance: Accreditation and the academy in an uncertain world. [special insert]. Briefing, 12(1), 1-5.

Leahy, M. J., Szymanski, E. M., & Linkowski, D. C. (in press). Knowledge importance in rehabilitation counseling. Rehabilitation Counseling Bulletin.

Leahy, M. J., Wright, G. N., & Shapson, P. M. (1987). Rehabilitation practitioner competencies by role and setting. Rehabilitation Counseling Bulletin, 31, 119-131.

Linkowski, D. C., & Szymanski, E. M., (1993). Accreditation in rehabilitation counseling: Historical and current context and process. Rehabilitation Counseling Bulletin, 37, 81-91.

Linkowski, D. C., Thoreson, R. W., Diamond, E. E., Leahy, M. J., Szymanski, E. M., & Witty, T. (1993). Instrument to validate rehabilitation counseling accreditation and certification knowledge areas. Rehabilitation Counseling Bulletin, 37, 123-129.

Longanecker, D. A. (1994). The new federal focus on accreditation. Academe, 80(4), 13-17.

Magner, D. K. (1993, April 14). Council on Postsecondary Accreditation Votes to Cease Operations. The Chronicle of Higher Education.

McGaghie, W. C. (1993). Evaluating competence for professional practice. In L. Curry, J.F. Wergin, & Associates, Educating professionals: Responding to new expectations for competence and accountability (pp. 229-261). San Francisco: Jossey-Bass.

McGuire, C. H. (1993). Sociocultural changes affecting professions and professionals. In L. Curry, J.F. Wergin, & Associates, Educating professionals: Responding to new expectations for competence and accountability (pp. 1-16). San Francisco: Jossey-Bass.

Mortimer, K. P., & Edwards, S. R. (1990). A president's view of the 1990s. New Directions for Higher Education, 70, 73-79.

Muthard, J. E., & Salomone, P. (1969). The roles and functions of the rehabilitation counselor. Rehabilitation Counseling Bulletin, 13, 81-168.

Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum.

Rees, N. (1993, July 29). Draft proposal on the successor structure of the Committee on Recognition: Memorandum to the COPA Community. Washington, DC: Council on Postsecondary Accreditation.

Rothman, R. A. (1987). Working: Sociological perspectives. Englewood Cliffs, NJ: Prentice-Hall.

Rubin, S. E., Matkin, R. E., Ashley, J., Beardsley, M. M., May, V. R., Onstott, K., & Puckett, F. D. (1984). Roles and functions of certified rehabilitation counselors. Rehabilitation Counseling Bulletin, 27, 199-224.

Skrtic, T. M. (1991). Behind special education: A critical analysis of professional culture and school organization. Denver: Love.

Szymanski, E. M., Linkowski, D. C., Leahy, M. J., Diamond, E. E., & Thoreson, R. W. (1993). Validation of rehabilitation counseling accreditation and certification knowledge areas: Methodology and initial results. Rehabilitation Counseling Bulletin, 37, 109-122.

Szymanski, E.M., Leahy, M. J., & Linkowski, D. C. (1993). Reported preparedness of certified rehabilitation counselors in rehabilitation counseling knowledge areas. Rehabilitation Counseling Bulletin, 37, 146-162.

Szymanski, E. M., & Leahy, M. J. (Eds.). (1993). Rehabilitation counseling credentialling: Research and practice [Special issue]. Rehabilitation Counseling Bulletin, 37.

Received: April 1994 Revised: September 1994 Acceptance: September 1994