The development and testing of APTA clinical performance instruments. (Research Report).

Physical therapist (PT) and physical therapist assistant (PTA) academic programs establish systems for evaluating students during their clinical experiences. Reasons for evaluation include, but are not limited to, determining whether the student's progress is satisfactory, assessing the student's readiness to enter practice, providing the student with feedback, and obtaining feedback on the education program relative to the currency, relevance, and application of content. (1,2) Various systems and instruments have been developed for these purposes. These are theoretically based on the principles of competency-based education. (3-5) Personnel at academic programs and clinical education sites have developed instruments, apparently with a goal of evaluating overall competence to practice as well as behavior specific to certain patient populations or clinical sites. (5-12)

Although many instruments had unique features, shared characteristics led some educators to seek consistency across educational programs (and students). Emergence of consortia, with personnel from multiple educational programs often working with a common group of clinical faculty, played a role in the development of uniform processes and instruments for assessing student clinical performance. (6,7,9,13) These trends, the task force believes, set the stage for developing the American Physical Therapy Association (APTA) PT and PTA Clinical Performance Instruments (CPIs). The purposes of this article are to describe the development of 2 CPIs (one for PTs and one for PTAs) and to provide information about the psychometric properties of the third drafts of these instruments (field study versions). This article does not report evidence about the psychometric properties of the final, published versions. No data are reported relating to the versions of the CPIs currently in use.

Background

In November 1993, a 10-person task force was charged by the APTA Board of Directors to develop clinical education evaluation instruments to measure student performance in PT and PTA clinical education. (14) Task force members were appointed by the Board of Directors from a group of 75 individuals with a variety of backgrounds, experiences, and expertise who were nominated by personnel from PT and PTA academic programs. The task force began its work by agreeing on a context in which they believe clinical education is provided. This was done in an effort to ensure that the resulting instruments could be used with minimal training, were reflective of current practice expectations, and met the needs of academic institutions to comply with accreditation criteria. (1,2)

The task force began its work in 1994 by agreeing on 3 foundation assumptions to use as a guide for the development of the CPIs: (1) that clinical competence is based on multiple behaviors deemed essential to the role of the PT or PTA, (2) that the CPIs should be constructed to measure performance along a continuum from novice to at least entry level, and (3) that the instruments must be responsive to the needs of both academic and clinical communities. (12,15) The task force believed that the design of the CPIs should allow measurement of behaviors, with multiple practitioners serving as educators. To achieve this goal, the task force attempted to develop instruments that could yield reliable scores as students provided patient/client care in a format appropriate for the environment to which they were assigned. Thus, the instruments would need to be psychometrically sound and provide useful information about student performance during clinical education. (16)

Method

The work of the task force and the evolution of the CPIs was in 4 phases that culminated in the development of instruments currently in use. These phases were: (1) development of the first drafts, including initial selection of target behaviors and a scoring/instructional protocol, (2) conduct of pilot studies using the second drafts of the PT and PTA CPIs created in response to feedback from a group of 50 people who the task force believed were experts in clinical education and research based on information provided on the nomination forms and in individual resumes, (3) testing of the third drafts to determine reliability, validity, and feasibility via field studies, and (4) modification of the third drafts in response to the field studies and preparation of the final versions for adoption by the APTA Board of Directors and for sale by APTA.

Phase I: First Drafts of the CPIs

The first drafts of the CPIs contained 23 PT and 20 PTA performance criteria and sample behaviors describing observable indicators for each performance criterion. Criteria were developed in an effort to be consistent with documents such as the first draft of A Normative Model for Physical Therapist Education, the Guide to Physical Therapist Practice, Volume I: A Description of Patient Management, (17) used by academic programs or clinical sites. A Normative Model for Physical Therapist Education describes practice expectations, educational outcomes, and content for the preferred PT curriculum. The Guide to Physical Therapist Practice describes the breadth and depth of PT practice, including patient or client management. Sample behaviors describing observable indicators for each performance criterion item were then identified. For example, in the first drafts of the CPIs, the performance criterion "Performs physical therapy treatment that achieves desired outcomes" included sample behaviors such as "performs treatment consistent with the plan of care," "provides treatment in a manner minimizing risk to the patient and others involved in the delivery of the patient's care," "adapts physical therapy treatment to meet the individual needs and responses of the patient," and "provides treatment in a manner minimizing risk to self."

A visual analog scale (VAS) was selected for educators to record the quality of observed behavior for each item on the CPIs. A horizontal line, 100 mm in length, was used to represent the continuum of points between the lowest level and the highest level of student performance that could be observed by a clinical instructor (CI). The line was anchored on the far left with the words "Novice Student Clinician" and on the far right with the words "Expert Clinician." A mid-range anchor was placed at 60 mm and was labeled with the words "Entry-Level Clinician." Use of the VAS as a recording format has been suggested to be appropriate when evaluating complex human performance that cannot (and perhaps should not) be divided into the type of discrete units of behavior easily recorded using other formats. (18,19) In addition, because continuous scales such as the VAS can reflect degree of change (20) better than categorical scales, (21) the task force members believed that the VAS met the goals of clinical education assessment more effectively than would other approaches. Because of the large number of possible ratings available with a VAS, it may also decrease problems associated with end aversion bias. (21) End aversion bias causes raters to avoid extreme rating categories. Because of end aversion bias, some authors (21) argue that a 5-point Likert scale might actually be used as a 3-point scale. There is a belief that loss of response categories tends to decrease both efficiency and reliability. (21) Members of the task force also felt that the use of a VAS would address the problem of raters adding plus or minus designations or decimals to categorical scales. This decision was based on the experience of task force members in using other instruments. Finally, the task force members feared that respondents may attach meaning to the numbers on a rating scale that are distinct from the verbal descriptors attached to those numbers. (21) The task force felt that this phenomenon was a likely occurrence in an academic setting, where numbers are often associated with grades and with student success or failure.

The first drafts of the CPIs included performance criteria that the task force considered essential, minimum elements for clinical practice. Four performance items related to safety and professional behavior were identified. The task force agreed that a problem with one of these items would be a warning or "red flag" for serious problems with student performance. The first drafts also included a preamble to provide users with a rationale for developing the CPIs, the basic assumptions upon which the instruments were designed, and reasons for considering its use. Directions for use of the CPIs also were included with the drafts.

The first draft versions of the CPIs were reviewed by 50 people within and external to physical therapy who, in the task force's opinion, possessed expertise in academic and clinical education, outcome assessment, evaluation, and psychometric test and measurements. Individuals who had been nominated but not appointed to the initial task force by APTA's Board of Directors formed the group. Feedback was gathered from this group on the structure of the instruments, clarity of directions, relevance and number of performance criteria, consistency between PT and PTA instruments, and mechanism for identifying problematic student performance on any single criterion. The group recommended that the task force clarify the directions in an effort to eliminate what the group perceived as ambiguity. The group suggested that the task force describe how assignment of grades would be made by an academic program. In addition, they suggested that the task force identify the purpose of sample behaviors and clarify that the list of sample behaviors was not exhaustive and provided a mechanism for assessing performance during midterm and final clinical education experience. They also suggested that PT and PTA instruments, where appropriate, be consistent to make it easier for the CI rater who may evaluate PT and PTA students at the same time in their clinical facility, using the same instrument.

Phase II: Second Drafts of the CPIs (Pilot Study Versions)

Modifications From the Previous Versions

The format of the CPIs was modified according to suggestions made by the group of 50 experts. The second drafts, or pilot study versions, of the CPIs included 23 PT and 20 PTA criteria (Tabs. 1 and 2). Two additional items (items 24 and 25 of the PT CPI and items 21 and 22 of the PTA CPI) were added to allow rating of the student's overall performance relative to academic and clinical expectations and entry-level performance. Particularly important performance criteria were labeled as "red-flag" items (Figure), and a symbol of a flag was added to the left of those performance criteria that were identified as such. A "Significant Concerns/ At-Risk" check box also was added to allow the CI to indicate at midterm or final evaluation when, in the estimation of the CI, a student's performance placed him or her at risk for failing the clinical experience (Figure). Additional CPI components included tables of contents to make it easier for the user to locate information within the CPI, glossaries to define terminology, and bibliographies to assist in user training that often occurs only through on-site materials reviewed by the CI. Members of the task force also added vignettes that depicted a hypothetical new graduate PT and a new graduate PTA who demonstrated competencies and deficiencies that might be observed in real life.

[FIGURE OMITTED]

Information Collected From Physical Therapy Community

In preparation for conducting the pilot studies of the PT and PTA CPIs, feedback was sought from potential users on the second drafts of the PT and PTA CPIs through a survey and the conduct of regional forums. Beginning in October 1995, second drafts of the CPIs were disseminated to 434 physical therapy academic program directors, 454 academic coordinators of clinical education (ACCEs), and their respective clinical education sites and CIs in the United States and Canada. Notification of the opportunity to review these draft instruments was provided through the Education Division newsletter, R.E.A.D. (December 1995), and PT Magazine (October 1995). Approximately 50 people requested copies of the CPIs.

Through the survey, the task force requested feedback on issues such as content, format, and process issues. Examples of issues raised included: how to distinguish performance requirements of the PT and PTA CPIs, whether the instruments were sufficiently comprehensive and user-friendly, how to mark the VAS to indicate student performance levels, how VAS scores can be converted into grades for use by academic programs, and clarity of the directions.

Ten APTA-sponsored regional forums were held between February 1996 and June 1996. These forums were presented by members of the task force for members of academic programs and consortia composed of groups of academic and clinical educators throughout the United States and in Victoria, British Columbia, Canada. During each forum, participants were asked semistructured questions to obtain verbal feedback, and they were surveyed in writing to obtain further comments and opinions. The survey instruments were distributed during the forum and were collected from participants at the completion of the forum. More than 700 people attended these forums. An estimated 350 additional people provided feedback and comments to APTA staff by mail on the written survey that accompanied the evaluation instruments, resulting in feedback from approximately 1,050 people.

Pilot Studies

Pilot studies were conducted on the second drafts of the PT and PTA CPIs between October 1995 and April 1996 in an effort to provide information on the internal consistency, construct validity and interrater reliability of these versions of the instruments as well as on the factor structure of the PT CPI. The task force considered such analyses necessary to refine the CPIs and to test the use of the CPIs in the clinical setting.

Procedure

Protection of human subjects. The University of Miami Medical Subcommittee for the Protection of Human Subjects reviewed the pilot study protocol. Because the researchers did not know the identity of the students or the CIs and the students were required to mail the data to APTA, thereby consenting to participate, this protocol was exempted from obtaining written informed consent from the students and CIs.

Instruments. In addition to the PT and PTA CPIs, several instruments were used in this study to examine characteristics and satisfaction of participants in the study. The Student Survey requested information about demographics, academic and clinical preparation, and satisfaction with the use of the CPIs. The CI Survey requested information on demographics, clinical setting, and satisfaction with the use of the CPIs. The ACCE Survey requested information on demographics and satisfaction with the use of the CPIs. User satisfaction was examined by having students, CIs, and ACCEs rate their level of satisfaction with various aspects of the CPIs, including time to complete, ease of use, and clarity of instructions. User satisfaction was measured using a 7-point Likert scale. A rating of 1 indicated that the respondent was "very dissatisfied," and a rating of 7 indicated that the respondent was "very satisfied."

Subjects. The sample size identified by members of the task force for conducting the pilot studies was limited to 350 PT students and 350 PTA students. This sample size was set by the task force because the group believed it was a reasonable target sample that could be obtained without knowing in advance how many programs would consent to participate and would have students on clinical education experiences during the time that the study was to be conducted. The sample size also was set in an effort to ensure a sufficient sample size for statistical analyses that would result in data that could be used to make decisions regarding the next draft versions of the CPIs. To obtain this sample, a 2-page questionnaire was mailed to all accredited and developing PT and PTA education programs to determine: (1) the willingness of the programs to participate in a pilot study of the PT or PTA CPI and (2) when students would be completing each of their clinical education experiences during the 1995-1996 academic year. Representatives of PT and PTA programs that could participate in the pilot studies and who had students completing their clinical education experiences between October 1995 and February 1996 were then contacted by telephone and asked to participate. Where multiple programs from the same state were able to participate, the task force considered obtaining a sample that had both public and private institutions, degrees of different levels, and varied levels of clinical education experience (eg, first, intermediate, and final). In addition, the length of clinical education experiences was considered.

Personnel from academic programs that agreed to participate in the pilot study identified, in the aggregate, 350 PT students and 350 PTA potential students willing to participate in the study. The pilot studies actually included 282 PT students from 31 US accredited PT professional education programs (9 baccalaureate and 21 master's degree) representing 26 states and 24 students from 2 PT education programs in Canada (Quebec and Ontario). Two hundred seventeen PTA students from 23 US accredited PTA education programs representing 17 states also participated in the pilot studies. Physical therapist students were first-year, second-year, and, in some cases, third-year students engaged in all levels of clinical education, including first, intermediate, and culminating clinical experiences. Likewise, PTA students were both first- and second-year students engaged in all levels of clinical education. Lengths of clinical experiences for PT and PTA students ranged between 1 and 9 weeks for both part-time (<35 hours/ week and [less than or equal to] 1 week) and full-time ([greater than or equal to] 35 hours/week and >1 week) experiences. A subset of pilot study participants, 70 pairs of PT CIs and 42 pairs of PTA CIs, each pair supervising one student, volunteered to be involved in the interrater reliability phase of the study.

On behalf of the academic programs, the ACCEs agreed in writing to identify possible student and CI subjects and to obtain verbal consent from those subjects to participate in the pilot study. The ACCE at each participating academic institution then obtained verbal agreement from students enrolled in the physical therapy program to participate in the study. When a student agreed to participate, the ACCE contacted the center coordinator of clinical education (CCCE) at the student's assigned clinical education site to determine whether the CCCE was willing to participate. If personnel at the clinical site were willing to participate, the CCCE obtained verbal agreement from the CI assigned to the student. If the CI agreed to participate, APTA's Department of Clinical Education was notified that a study pair (one supervising CI and one student) was available at that clinical education site during a specified time period. For the interrater portion of the study, clinical sites were identified where a second CI was available and willing to evaluate the student's performance without consulting with the PT who supervised the student.

The CIs and students were instructed to first complete the instrument that was typically used by the academic program for midterm and final evaluations to assess the students' performance. Once this was done, the CIs and students also completed the CPI for both the midterm and final evaluations. The students and CIs were instructed to compare their midterm and final evaluations on the CPI. In the interrater reliability portion of the study, both CIs were instructed to independently conduct midterm and final evaluations using the CPI and to not discuss or compare results. Neither the students nor the CIs were instructed to attach numbers to the vertical marks on the VAS. After the final evaluation was completed, the students and the CIs independently completed their respective survey questionnaires and placed both copies of the CPI and their survey questionnaires in stamped envelopes addressed to the researchers. The students then sealed the envelopes and had the prerogative to and responsibility for placing the completed CPIs and survey questionnaires in the mail. The students and CIs were linked as a pair through their identification numbers, but neither the students nor the CIs could be individually identified.

Upon receiving the completed CPIs, staff in APTA's Department of Clinical Education copied the CPIs completed by the CIs and used the code to identify the clinical education sites and academic institutions. Staff then forwarded copies of the completed CPIs from the supervising CIs to the ACCEs at the students' academic institutions. The ACCE Survey was included. Once the ACCEs received copies of the CPIs from all of their academic program's participating students, they were supposed to complete and return the ACCE Survey. Once all CPIs had been copied and forwarded to ACCEs, the master list was destroyed.

Analysis. For purposes of our study, the marks on the VAS were measured. The zero on the ruler was aligned with the lower anchor of the VAS such that scores ranged from a possible 0 to 100 mm for each scored CPI item. Data entry was performed at the Division of Physical Therapy, University of Miami. All data were managed and analyzed using a VAX mainframe computer and SAS version 6.1 * statistical software. (22) Analyses of internal consistency, interrater reliability, validity, and user satisfaction were conducted. Internal consistency, in the view of the task force, represents a psychometric characteristic of instruments that are designed to measure one overall construct. In the case of the CPIs, that construct is performance as a PT or PTA. If an instrument is designed to measure one construct, then all items of the instrument should be related to that construct. Internal consistency measures the degree to which items are related. (23,24) A total score was generated by taking an average of the item scores.

Internal consistency. The purpose of the CPIs is to measure quality of behavior, namely the ability of a student to perform as a PT or PTA. Conceptually, these levels of performance range from a novice level to an entry level. If an instrument is designed to measure various aspects of a single behavior (performance as a PT or PTA), then it is desirable that all the items included in the instrument measure different features of that behavior rather than different parts of dissimilar behaviors. (21) Internal consistency of the PT and PTA CPIs was examined by calculating Cronbach alphas.

Interrater reliability. Reliability can be demonstrated for a measurement only when the measurement is applied to a specific population. (21) Interrater reliability was selected by the task force for analysis in this study because the task force was most interested in whether 2 raters could agree about the performance of students at a given time. Interrater reliability of the CPIs was examined by calculating type 2,1 (23) intraclass correlation coefficients (ICCs) for each of the items and for a total score generated by taking an average of the item scores.

Construct validity. Two hypotheses were generated to determine construct validity. The first was based on the belief that students' performance should differ between students on first clinical experiences and final clinical experiences. Evidence for construct validity was examined by calculating a Student t test to compare total CPI scores for students at the end of their first clinical experience with total CPI scores for students at the end of their last clinical experience. In the second hypothesis, the task force also assumed that clinical performance of PT and PTA students should be related to the amount of prior clinical experience. Therefore, construct validity was examined by calculating Pearson correlation coefficients to determine the relationship between the CPI item scores and total days of clinical experience. Criteria for statistical significance were set at .002 to correct for multiple tests.

User satisfaction. User satisfaction with the CPIs was measured on a 7-point Likert scale, applied to 17 items, with 1 indicating that the respondent was "very dissatisfied" and 7 indicating that the respondent was "very satisfied." The median level of satisfaction with all 17 aspects of using the instruments was calculated separately for PT and PTA students, for PT and PTA CIs, and for ACCEs. Median levels of satisfaction were calculated because the calculation of medians rather than means is appropriate for ordinal data.

Principal components analysis. A principal components analysis was performed to explore the number of distinct constructs represented in the CPIs. Because of the sample sizes, only data from the PT CPI were used for the analysis, and the PTA CPI was not studied in this way.

Results of the Pilot Studies

PT CPI internal consistency. The Cronbach alpha for the PT CPI was .97, indicating a high level of internal consistency.

PT CPI interrater reliability. Reliability estimates for the items on the PT CPI ranged from -.02 to .62, with the majority of items demonstrating what the task force would consider moderate reliability (Tab. 1).

PT CPI validity. Pearson correlations between length of prior clinical experience and several items, including "Designs a physical therapy plan of care that integrates goals, treatments, and discharge plan," were .49. The lowest correlation between length of prior clinical experience and a CPI item was -.05 for the global rating item "Rate this student's overall performance relative to academic and clinical expectations." This result was anticipated by the task force because this item essentially asks the rater to adjust for prior clinical experience. This same item failed to demonstrate a difference in score between students on their first and final clinical experiences. All other items demonstrated differences between students on the first and final clinical experience.

PTA CPI internal consistency. The Cronbach alpha for the PTA CPI was .96, indicating a high level of internal consistency.

PTA CPI interrater reliability. Reliability estimates for the items on the PTA CPI ranged from .35 to .89 (Tab. 2). This level of reliability was much higher than the task force anticipated and led the group to have some concerns about the independence of the 2 raters.

PTA CPI validity. As was true for the PT CPI, the correlation was very small for the PTA CPI (.15) between length of prior clinical experience and the global rating item score for the item "Rate this student's overall performance relative to academic and clinical expectation." The correlation between length of prior clinical experience and a PTA CPI item score was much higher (.39) for items such as "Makes clinical decisions within the scope of PTA practice" and "Participates in modifying the plan of care." There were differences in average CPI scores between students on first and final clinical experiences for all the PTA CPI items.

Principal components analysis. The unrotated principal components analysis of the PT CPI produced 2 components, with the first component accounting for 19% of the variance and the second component accounting for only 2% of the variance. All 25 items of the CPI loaded more strongly on the first component. A Varimax rotation was then performed. The first component accounted for 12.1% of the variance, and the second component accounted for 8.7% of the variance, suggesting that the instrument might contain 2 constructs. Component 1 was defined by items 6, 8 through 20, 23, and 25, and component 2 was defined by items 1 through 5, 7, 21, 22, and 24. Items associated with the first construct represented physical therapy-specific clinical skills required for practice such as the physical therapy examination, diagnosis, plan of care, and treatment. The items associated with the second construct were less specific to physical therapy in that they represented behaviors required in all areas of clinical practice and applicable across all patient types. These items included such general clinical behaviors as safe practice, ethical behavior, professional behavior, and cultural considerations (Tab. 3). However, all items showed moderate to strong loading on both factors, with corresponding high communality estimates, suggesting that the PT CPI is probably measuring one underlying construct. The only exception to this finding was for the global rating item "Rate this student's overall performance relative to academic and clinical expectations." This item had a communality estimate of only .23, probably reflecting the fact that it is the only item that rates overall or global performance relative to academic and clinical expectations rather than a specific performance criterion associated with a component of entry-level practice.

Limitations of the Pilot Study

The students, CIs, and ACCEs who participated in the pilot study were not trained in the use of the CPI or in the study protocol, other than being given written instructions included in the study protocol and the directions included with the CPI. Although these written instructions and directions described the use of the VAS, explained the distinction between criteria and behavioral criteria, and cautioned the evaluator about the risks of rater bias and inconsistency, reliability estimates may have been adversely affected by the lack of training and control. Although training is now available on the CPI, one cannot assume that all CIs using the CPI have received such training. Therefore, in those situations where training is not available or has not been provided, the situation under which the CPIs were tested may mimic the real world. Because of the design of the study, the researchers were unable to determine whether the subjects read, understood, or followed the study protocol or directions to complete the CPI. (25) The task force acknowledges that wording and terminology problems could cause some items to be less reliable, but also that large-scale personal training of CI evaluators is not feasible. Data generated from this study were used to compare the relative performance of various items based on our assumptions that any methodological difficulties would affect all items equally. Therefore, items with lower reliability coefficients were examined for possible problems with wording and terminology.

Results of the pilot studies along with input from regional forums and other focus group meetings were used to revise the second versions of the CPIs. In addition, the task force's experience in working with raters, feedback from a community of interest (ie, academic faculty and clinical educators), and analyses from the pilot studies were used to design and conduct the field studies.

Phase III: Third Drafts of the CPls (Field Study Versions)

Modifications From the Previous Versions

Information from PT and PTA academic faculty and researchers, PT and PTA clinical educators, PT and PTA students; data from the pilot study; and consultation provided by a psychometrician were used by the task force to make changes to the PT and PTA CPIs. In the third drafts of the CPIs, the performance criteria were expanded, with the PT CPI increasing to 24 items and with the 20 PTA performance criteria retained (Tabs. 4 and 5). This draft introduced a modified VAS that eliminated the "Expert Clinician" anchor at the far right as used in the first and second versions. The result was the inclusion of 2 anchors on the VAS; the far left anchor was labeled "Novice Clinical Performance," and far right anchor was labeled "Entry-Level Performance." The task force chose to make this modification based on (1) feedback offered by PT and PTA academic faculty and clinical educators about the likelihood that students would (or should) achieve entry-level performance status and (2) advice from the consultant that multiple VAS anchors may complicate or confuse rater behavior. Some physical therapy educators argued that a mechanism was needed to enable acknowledgment of excellence in student performance. Thus, a "With Distinction" box was added to recognize student performance that exceeded entry level on any criterion (Figure).

Wording changes in the performance criteria and instructions were made in an effort to enhance clarity and consistency with APTA documents, including the Evaluative Criteria for the Accreditation of Education Programs for the Preparation of Physical Therapists, (1) Evaluative Criteria for the Accreditation of Education Programs for the Preparation of Physical Therapist Assistants (2) third draft version of A Normative Model for Physical Therapist Professional Education (to evolve into A Normative Model of Physical Therapist Professional Education: Version 97 (26)), and preliminary findings from a consensus conference convened to develop A Normative Model for Physical Therapist Assistant Education: Version 98. (27) Further modifications included adding new terms and their definitions to the glossary, clarification and expansion of the sample behaviors, clarification of the performance criteria and user instructions, and refinement of the case vignettes. New additions to this version included a box to indicate that the entire performance criterion was "Not Observed" (Figure) and an area for student and evaluator signatures.

Information Collected From the Physical Therapy Community

Availability of third draft versions of the CPIs were announced in the December 1996 issue of PT Magazine, the November 8, 1996, issue of PT Bulletin; the November 1996 Education Division newsletter, R.E.A.D.; and the Fall 1996 Education Section newsletter for review and comment by request. Approximately 80 copies of these instruments were requested for review. In addition, third draft versions of the CPIs were mailed to 493 physical therapy academic program directors, 516 ACCEs, and their respective clinical education sites and clinical educators throughout the United States and Canada. In addition, task force members presented the third draft versions of the CPIs at several national forums, which allowed nearly 500 people to provide feedback. For academic faculty and clinical educators who wanted to discuss the instruments at the local or regional level, the task force developed a questionnaire that was distributed with the instruments. This questionnaire posed closed-ended questions designed to obtain information about respondent demographics, opinions about whether performance criteria reflected entry-level expectations, whether sample behaviors accurately described each performance criterion, usefulness of the VAS in documenting student performance, ability to document excellence in student performance beyond entry-level expectations, ability to identify student problems in clinical performance, and management of student performance when the "Significant Concerns/At-Risk" box is checked or when the performance criterion is "red-flagged" or "non-red-flagged." In addition, several open-ended questions were posed to: (1) request respondents to identify criteria that need to be revised, added, or deleted, (2) identify major advantages and disadvantages of the instruments, (3) identify questions or concerns, and (4) solicit any comments that would assist in refinement of the instruments. People reviewing the instruments were encouraged to complete and submit the questionnaire for use in development of the fourth (final) versions of the CPIs.

Field Studies

Field studies were conducted between October 1996 and July 1997 to assess the psychometric properties of the third versions of the instruments. The field studies examined: (1) internal consistency, (2) interrater reliability, and (3) construct validity of the PT and PTA CPIs. The University of Miami Medical Subcommittee for the Protection of Human Subjects reviewed the field study protocol. Again, because of the subject anonymity guaranteed by the design, this protocol was exempted from informed consent.

Procedure

Instruments. In the field study, as in the pilot study, the research was designed to test the internal consistency, interrater reliability, construct validity, and student, CI, and ACCE satisfaction with the third drafts of the PT and PTA CPIs. The field study also examined the discriminant validity of the CPIs by examining the relationship between CPI item scores and social competence as measured by the Social Skills Inventory (SSI). (28) The SSI was administered to a subset of 50 PT students and 50 PTA students to determine whether the CPI was measuring social skills rather than entry-level clinical performance. The researchers believed that although social competence is a desirable characteristic in a clinician, it is not the same as the ability to perform as a PT or PTA. Therefore, the task force expected little or no relationship between SSI score and item scores on the CPI.

Identification of this subset of students to complete the SSI was determined by contacting the academic programs participating in the study to inquire whether their students participating in the field studies were willing to complete the SSI. Additionally, programs were selected if students could be proctored by a faculty member while completing the SSI and agreed to return the completed SSIs to the task force. The SSI was selected to be included in the field studies because it had been shown to be a reliable and valid measure of social competence not restricted to any particular professional situation. (28)

The SSI (28) is a 90-item research instrument that has been designed as a self-report measure of social communication skill in 6 domains that, when combined, comprise global social skill or social competence. Social competence is defined as follows: "... a multidimensional construct that includes skills in receiving, decoding, and understanding social information. It further involves social participation skills such as verbal and emotional expression, regulation of social behavior, and social role playing abilities." (29) The SSI is designed for use with adults at an eighth-grade reading level or above. By use of a 5-point Likert scale, the SSI can be used to measure communication skills on 2 levels--emotional and social skills. Communication skills are measured in 6 domains: emotional expressivity (EE) measures the skill with which individuals communicate nonverbally; emotional sensitivity (ES) measures skills in receiving and interpreting the nonverbal communication of others; emotional control (EC) measures ability to control and regulate emotional and nonverbal displays; social expressivity (SE) assesses skill in verbal expression and the ability to engage others in social discourse; social sensitivity (SS) assesses the ability to interpret the verbal communication of others; and social control (SC) assesses skill in role playing and social self-presentation.

The SSI yields a score that is supposed to reflect overall social competence. The majority of subjects who participated in initial validity studies of the SSI (29) were under-graduate and graduate college or university students from multiple disciplines, including the health care professions. Intraclass correlation coefficients for test-retest reliability of the SSI scores ranged from .81 to .96 on a small sample of subjects (N=40) who completed the SSI twice with a 2-week interval between tests to test the stability of scores on attributes that should not vary much within that period of time. The SSI scores demonstrated convergent and discriminant validity in a series of validity studies reported by Riggio. (29) Factor analysis, based on a sample of 629 undergraduate students and conducted using a 6-factor solution and the principal axis method with Varimax rotation, revealed that all 6 of the predicted factors emerged (28(p8),29) in the 6 domains.

Subjects. Subjects volunteered and were identified using the same procedures as those used in the pilot studies except that there were no limitations placed on the number of PT and PTA programs, CIs, and students who could participate. Students participating in the field studies had to complete their clinical education experiences between October 1996 and May 1997. Instructions provided to the CIs and students were the same as those provided in the pilot studies. For the interrater portion of the study, 35 groups for the PT students and 35 groups for the PTA students were identified for which a second CI was available and willing to participate. In addition, participating students, CIs, and ACCEs completed surveys on subject characteristics, academic and clinical preparation, and satisfaction with the use of the CPIs.

A subset of 50 PT students and 50 PTA students who participated in the field studies also completed the SSI. (28) Once coded, data were to be forwarded to APTA's Department of Clinical Education for subsequent scoring and analysis.

Subject demographics. Representatives of 44 PT academic programs and 27 PTA academic programs agreed to participate in the field studies. Of the PT programs in the United States, 54.5% (n=24) were from public institutions and 45.5% (n=20) were from private institutions. Twenty-eight (63.6%) of the PT programs awarded professional master's degrees, and 16 (36.3%) of the PT programs awarded professional baccalaureate degrees. Programs were from all geographic regions of the United States, with PT programs located in 27 states and PTA programs located in 20 states. In the aggregate, physical therapy academic programs were located in 35 different states. In addition, there were 2 physical therapy programs in Quebec and Toronto, Canada (one representing a public institution and one representing a private institution), with both awarding the professional baccalaureate degree.

Three hundred clinical education sites were involved in the PT study, and 225 clinical education sites were involved in the PTA study. They were in rural, suburban, and metropolitan locations. Physical therapist student clinical experiences ranged from part-time (<35 hours per week) clinical experiences of 2 to 5 weeks in length to full-time ([greater than or equal to] 35 hours per week) clinical experiences of 2 to 12 weeks in length. These experiences included first, intermediate, and culminating clinical experiences. Physical therapist assistant clinical experiences ranged from part-time to full-time clinical experiences of 2 to 10 weeks in length (Tab. 6).

In the aggregate, participating CIs were providing care to patients across the life span, at various levels of disease acuity, and in a broad ranges of venues, including school systems, inpatient care and outpatient facilities, home health care settings, private practices, subacute rehabilitation settings, government-based facilities, industry, and extended care and nursing facilities.

Analysis. Data from the field studies were analyzed using the same approach as was used in the pilot studies. Data entry was performed at the Division of Physical Therapy, University of Miami. All data were managed and analyzed using a VAX mainframe computer and SAS version 6.1 statistical software. (22) As in the pilot studies, all VAS items were measured and assigned a number from 0 to 100 mm. The "With Distinction" box was not used in calculating the numerical score. "Red-flag" items were scored in the same manner as other items. Some experts contend that interrater reliability of individual items is typically lower than the interrater reliability of a composite score. (21) Interrater reliability of a composite score was determined by first calculating an average score for all items rated by both raters, excluding the items requiring judgment of overall performance. The reliability between these 2 composite scores was examined using an ICC (2,1). One additional analysis of validity was performed based on the assumption that item scores on the CPI should be unrelated to a student's social skill or social competence. Pearson correlation coefficients were calculated to determine the relationship between CPI item scores and SSI total score. The criteria for statistical significance were set at .0002 to correct for multiple tests.

Interrater reliability. Reliability coefficients (ICCs) for scores for items on the third draft of the PT CPI ranged from .21 to .76. The lowest reliability coefficient was found for scores for 2 items (25 and 26) measuring overall performance. No ICC could be calculated for item 17 due to a low response rate for that item (Tab. 7). Interrater reliability coefficients for scores of items on the PTA CPI ranged from -.12 to .78. For the PTA CPI, the items with the highest reliability were items 21 and 22, which were designed to measure overall performance (Tab. 8). The ICCs for interrater reliability were .87 for the PT CPI total score and .77 for the PTA CPI total score.

Results of the Field Studies

Descriptive statistics. One hundred eighty-one PTA students and 319 PT students participated in the field studies. Slightly more than 30% of both groups were male. The mean age of the PTA students was 29 years (SD=7.5, range=19-51), and the mean age of the PT students was 26.7 years (SD=5.5, range=20-47) (Tab. 6). The mean age of the PT students' CIs was 33.4 years (SD=7.6, range=23-63), with an average of 6.2 years (SD=5.4, range=0-30) of experience as a CI. The PTA students' CIs were similar to the PT students' CIs, with a mean age of 35.3 years (SD=8.3, range=22-64) and an average of 5.8 years (SD=8.3, range=0-30) of experience as a CI. Approximately one quarter of both groups of CIs were male. On average, the months of prior course work and the length of the clinical experience were longer for PT students than for PTA students (Tab. 6). For both the PT and PTA versions of the CPI, the mean item scores were all above 80, although the range of scores was greater than 70 for all items except for item I on the PT CPI and items 5 and 21 on the PTA CPI (Tabs. 4 and 5). The proportion of respondents marking an item "Not Observed" varied widely. Responses ranged from a low of 0% for item 1 ("Practices in a safe manner that minimizes risk to patients, self, and others") to a high of 63% for item 17 ("Provides consultation to individuals and outside organizations"). The use of "Not Observed" for the PTA CPI was also highly variable. Responses ranged from 63% for item 6 ("Communicated in ways that are congruent with situational needs") to 55% for item 2 ("Assists the PT in addressing prevention, wellness, and health promotion needs of individuals, groups, and communities") (Tabs. 4 and 5).

Internal consistency. The PT version of the CPI had a Cronbach alpha of .97, and the PTA version of the CPI had a Cronbach alpha .96.

Known groups method validity. Data from 310 PT students and 173 PTA students were used to analyze the relationship between CPI item scores and prior clinical experience. Data from a subset of 68 PT students and 89 PTA students who were on either their first or final clinical experience during the field study were used to perform the known groups analysis. Data from a subset of 31 PT students and 39 PTA students were used to analyze the relationship between CPI item scores and SSI total scores. The actual number of subjects for each analysis varied across CPI items based on the use of the "Not Observed" option. The majority of items on both the PT and PTA versions of the CPI demonstrated differences between students on their initial and final clinical experiences (Tabs. 9 and 10). However, there were some exceptions to this for the PT CPI. Although the means for the initial and final clinical experience groups were different, items 2 ("Presents self in a professional manner"), 17 ("Provides consultation to individuals and outside organizations"), 18 ("Addresses patient needs for services other than physical therapy"), 20 ("Incorporates an understanding of economic factors in the delivery of physical therapy services"), 21 ("Utilizes support personnel according to legal and ethical guidelines"), and 24 ("Addresses prevention, wellness, and health promotion needs of individual, groups, and communities") and global item 25 ("Rate this student's overall performance relative to academic and clinical expectations") were not different using the conservative criterion of P=.0002.

Convergent validity. Pearson correlations between CPI item scores and total days of clinical experience ranged from .12 (P=.0317) to .40 (P=.0001) for the PT version of the CPI and from .12 (P=.1000) to .34 (P=.0001) for the PTA version of the CPI (Tabs. 9 and 10). Item 25 on the PT CPI and item 21 on the PTA CPI ("Rate this student's overall performance relative to academic and clinical expectations") were not correlated with total days of clinical experience. Several items on the PTA CPI were not correlated with amount of prior clinical experience (Tab. 10). Items not correlated included items 2 ("Conducts self in responsible manner"), 4 ("Adheres to ethical practice standards"), 8 ("Adapts delivery of physical therapy care to reflect respect for and sensitivity to individual differences"), 16 ("Manages resources to achieve goals of the practice setting"), 19 ("Formulates and implements a self-directed plan for career development)", and 20 ("Assists the PT in addressing prevention, wellness, and health promotion needs of individuals, groups, and communities") and global item 22 ("Rate this student's overall performance relative to entry level").

Discriminant validity. Pearson correlations between the SSI total score and PT CPI item scores were consistently very low and failed to reach statistical significance. None of the items on the PTA version of the CPI were correlated with total SSI score (Tab. 10).

User satisfaction. Physical therapist and PTA students, CIs, and ACCEs rated their satisfaction with the use of the third drafts of the CPIs on a 7-point Likert scale. Data, in the task force's view, were ordinal; therefore, medians were reported. Respondents were generally satisfied (5 or 6) with most aspects of using the CPIs, although many items produced essentially neutral (4) median responses (Tab. 11). All users consistently rated the CPIs with a satisfaction score of 5 for ability to distinguish pass/fail, comprehensiveness, and identifying the nature of a student problem and a satisfaction score of 4 for identifying weaknesses in academic curriculum.

Physical therapist CIs appeared to have the highest overall level of satisfaction (6) with the CPIs, especially relative to the clarity of instructions for comments and use of the VAS and the value of using "red-flag" criteria. The ACCEs reported the highest level of satisfaction (6) with the CPIs for the value of using "red-flag" criteria as a basis for giving feedback, identifying a student problem, and usefulness of performance criteria examples. The only item responded to by the ACCEs with a median level of satisfaction less than neutral (4) was the use of the CPIs "as a basis for grading." This item produced a median satisfaction score of 3, representing mild dissatisfaction. Physical therapist and PTA satisfaction ratings were similar (5 or 4) for all items.

Discussion of Field Study Findings

Internal consistency. The purpose of the CPIs is to measure quality of behavior, namely clinical performance, as a PT or PTA. The level of item homogeneity of an instrument was measured using the Cronbach alpha. This statistic ranges from 0 to 1, with I indicating perfect item homogeneity. (21,23) Both the PT and PTA versions of the CPI demonstrated very high levels of item homogeneity (.97 and .96, respectively). This finding strongly suggests that the items in both versions of the CPI measure various aspects of a single behavior.

Reliability. Reliability coefficients (ICCs) for individual items ranged from .21 to .76 on the PT CPI and from .00 to .78 on the PTA CPI. The reliability of the total score yielded ICCs ranging from .50 to .75 for the PT and PTA versions of the CPI. These data support the notion that the easiest way to increase the reliability of a score is to increase the number of items contributing to the score. (21,23) As long as the correlation between items is not 1.0, increasing the number of items, in our view, will increase an instrument's ability to be used to distinguish among individuals much more than it will increase measurement error, thereby theoretically resulting in improved reliability. Likewise, the task force believes that this explains why a single-item measure such as "Is this student ready to practice as a physical therapist?" is much less reliable than a 25-item index measuring various aspects of readiness to practice. Scores for most items on the third draft of the PT CPI had better reliability than did scores for comparable items on the second draft version of the PT CPI (Tabs. 1 and 7). This would tend to indicate that the changes the task force made to improve clarity of the instrument were warranted. The results for the PTA CPI are much less clear. The interrater reliability for the second draft version of the PTA CPI, in our opinion, was unexpectedly good (ICC [2,1] above .75). By comparison, the interrater reliability for the third draft version was much worse, even for items that had not been altered (Tabs. 2 and 8).

Construct validity. Construct validity reflects the ability of an instrument to measure an abstract concept, (21,23) the conceptual (theoretical) basis for using a measurement to make an inferred interpretation. (30(p61)) In the case of the CPIs, the construct being measured was clinical performance. A number of approaches were used to examine construct validity. The known groups method (23) was used to examine construct validity by comparing a group that should have high levels of the construct (students at the end of their final clinical experience) with a group that should have low levels of the construct (students at the end of their first clinical experience). The task force anticipated that these 2 groups would differ on all items on the CPI and would likely differ on some performance criteria more than others. For example, on the one hand, item 2 on the PT CPI ("Presents self in a professional manner") is an aspect of clinical performance that the task force believed should be mastered early in a student's clinical education. Thus, the task force expected and found no difference in scores for this criterion between students at the end of the first and final clinical experience for this item. On the other hand, items 10 through 15 on the PT CPI represent aspects of clinical performance that should continue to improve until the end of the final clinical experience. For these items, the task force found differences ranging from 23.4 to 31.9 between the scores of students at the end of the first and final clinical experiences. For, items 17, 18, 20, 21, and 24, there were no differences between students on the first and final clinical experiences. These were the same items that were marked "Not Observed" by a large proportion of the participants in the field studies. Some of these items such as item 24 ("Addresses prevention, wellness, and health promotion needs of individuals, groups, and communities") represent what the task force considered aspects of PT practice that are currently demonstrated less frequently but are presumed to become more prevalent in the future. Without formal training in the use of the CPI, the CIs in this study may have had some difficulty rating these items relative to entry-level practice.

Concurrent validity. Concurrent validity demonstrates construct validity by showing that 2 measures that should be related are highly correlated. (23) The task force hypothesized that readiness to practice should be related to the amount of clinical experience students had at the time the CPIs were completed. The task force contends that amount of clinical experience was an important factor, but certainly not the only one, explaining readiness to practice. Therefore, the task force expected a moderate correlation between total days of clinical experience and scores on CPI items. In addition, the task force believed that clinical experience would produce a greater change in some behaviors than in others.

The task force hypothesized that items that changed more in response to experience would correlate more strongly to days of clinical experience than would other items. This pattern emerged through the data analysis. For the PT CPI, all items were correlated to days of clinical experience. Items such as items 7 ("Produces documentation to support the delivery of physical therapy services") and 14 ("Performs physical therapy interventions in a technically competent manner") were more strongly correlated to days of clinical experience than were items such as items 3 ("Demonstrates professional behavior during interactions with others") and 5 ("Adheres to legal practice standards"). A similar pattern was seen for the PTA CPI in that items such as items 11 ("Participates in modifying the plan of care") and 12 ("Performs physical therapy interventions in a technically competent manner") were correlated with days of clinical experience. In contrast, items such as item 2 ("Conducts self in a responsible manner" or item 4 ("Adheres to legal practice standards") were not correlated with days of clinical experience.

Discriminant validity. The task force assumed that although social skills are a desirable trait in a PT or PTA, they are not the same as clinical performance as a PT or PTA. An individual could have excellent social skills but, unless educated as a PT or PTA, he or she would not have demonstrable physical therapy clinical skills. Therefore, the task force believed that if the CPIs measured clinical performance as a PT or PTA rather than social skills, there should be little if any correlation between CPI scores and the total score on the SSI. This was true for the PT CPI.

User satisfaction. As is often the case with changes in a measurement tool, there can be some discomfort among users. The task force believed the CPIs would be no exception because they differed in many ways from student clinical performance instruments that were in use. As a result, the task force felt it was important that all groups of respondents (students, CIs, and ACCEs) were satisfied with the CPIs, particularly the clarity of instructions for comments and the use of the VAS. However, both PT and PTA students and PT and PTA CIs indicated on the user-satisfaction survey a neutral (4) median level of satisfaction with the time required to complete the CPIs (Tab. 11). This response, at least in part, partially reflects the time demand associated with learning to use a new instrument. The task force believes it also reflects time constraints confronting most clinicians involved in providing high-quality student clinical education. Additional training and experience with using the CPIs, in our view, should decrease the amount of time required to complete the instruments and could improve satisfaction.

The only response where median satisfaction was less than neutral was for ACCEs using the CPIs "as a basis for grading." Because assigning grades for clinical experiences is the responsibility of the academic institution, only the ACCEs actually use the CPIs for this purpose. The third drafts of the CPIs used during the field studies contained no guidelines for how to use the CPIs to generate a score or grade. Grading remains a prerogative of the academic program, which must determine what average, total, or pattern of item scores is required to pass. Given the wide variety of curricula, developing guidelines for grading was clearly beyond the scope of the field studies and the role of the task force. In addition, it would infringe on institutional prerogatives for determining grades.

Decision-making process. The process used by members of the task force to develop the instruments was complex and multifaceted. The instruments needed to be psychometrically sound, but they also had to be responsive to the needs of users (ie, physical therapy academic faculty, ACCEs, clinical educators, and students). Two examples are provided to illustrate this decision making, the first where members of the task force based their decision on data and the second where the prevailing decision was based on the needs of users.

In this first example, a decision regarding the inclusion of overall performance ratings in the CPIs was determined based on the results of the field study data. The task force believes student evaluation instruments routinely include an item that asks evaluators to rate the student's overall or global performance. Likewise, the CPIs included a global rating in the first 3 drafts. However, based on the field studies, results indicated that global ratings were not as reliable as those items that measured specific levels of performance. Even though users were accustomed to making a global performance rating and personnel at academic programs were familiar with assessing such a rating, the evidence did not support the continued inclusion of this type of rating. As a result, in the fourth (final) versions of the CPIs, the global ratings were excluded.

A second example illustrates how a decision to revise the CPIs was made in response to the feedback from users. The instruments were developed as outcome measures, and inclusion of a comprehensive list of interventions as a checklist for item 14 of the PT CPI and item 12 of the PTA CPI was not practical. However, users were familiar with and voiced their preference for including a list of interventions (PT) or technical skills (PTA). These are found in other instruments and help to guide evaluations of student competence. As a result, an appendix was added to the PT CPI that listed the categories of interventions as described by the Evaluative Criteria for the Accreditation of Education Programs for the Preparation of Physical Therapists, (1) and an appendix was added to the PTA CPI that delineated the entry-level technical skills (performance of all or a component of the skill) as found in A Normative Model of Physical Therapist Assistant Education: Version `98. (27) Thus, the resultant changes made to the third drafts of the PT and PTA CPIs to create the fourth (final) versions reflected decisions that were both data based and, in our view, responsive to physical therapy academic faculty, ACCEs, clinical educators, students, and researchers who had provided ongoing feedback and comments in support of the development of the evaluation instruments.

Limitations of the Field Studies

The quasi-experimental design of the field studies created a number of problems, especially for the reliability phase of the field studies. The lack of direct contact between the researchers and the subjects meant that the researchers were unable to determine whether the subjects read, understood, or followed the study protocol or directions to complete the CPI. In some ways, however, this may mimic what could occur if the CPIs were purchased and used without training, because again the researchers would not be able to train users of the instruments. However, the task force recommends, as with any instrument, that the final versions of the CPIs should be used by CIs after receiving training. This lack of control was the direct result of a design that was intended to protect the identity of the participants in an effort to minimize response bias due to social desirability and to eliminate the need for informed consent of students and CIs. Social desirability is the tendency of a research subject to respond in the way he or she thinks the researcher would want him or her to respond. (21) The risk of social desirability bias is increased as contact between the subject and the researcher is increased. Lack of contact and anonymity was done in an effort to ensure that the CIs were free to candidly evaluate the student without having to consider the possibility that the student or CI could be identified and be embarrassed by having their scores revealed to members of the task force or APTA staff.

Because there was no direct contact between the researchers and the participants, it was impossible to ensure that the 2 CIs participating in the reliability study had understood and followed the directions for using the CPI, performed their ratings independently, observed the student during the same period of time, or had sufficient observations with which to rate the student.

Likewise, the task force believes a lack of training probably also contributed to what we feel was an overuse of the "Not Observed" category. The task force believes that all items on both versions of the CPI can be rated in almost any setting. The task force contends that lack of understanding of the sample behaviors included under each item may have caused some CIs to use the "Not Observed" category more often than expected or needed. Most items were scored on the VAS in the pilot studies when the "Not Observed" category was not available as an option for completion on the CPI. However, with the introduction of the "Not Observed" category in the third drafts of the CPIs, the majority of CIs marked "Not Observed" for at least one item during the field studies. For example, on the PTA CPI, item 20 "Assists the PT in addressing prevention, wellness, and health promotion needs of individuals, groups, and communities" was marked "Not Observed" by 55% of raters in the field study; whereas only 29% of raters did not score the VAS for this item during the pilot study. For the PT CPI, item 17 "Provides consultation to individuals and outside organizations" was marked "Not Observed" by 63% of raters in the field study, whereas 49% of raters did not score the VAS for this item during the pilot study. As a consequence, for the field studies, it became inappropriate to use principal components analysis to examine construct validity because the data for participants with one or more "Not Observed" items were dropped from the analyses. The remaining sample was too small to perform a principal components analysis on a 24-item instrument.

Another limitation associated with the "Not Observed" responses may have been the use of A Normative Model of Physical Therapist Professional Education (27) and a draft version of A Normative Model of Physical Therapist Assistant Education. These are consensus-based documents that describe preferred expectations for entry-level practice. Performance expectations for entry-level practice described in these models include knowledge, skills, and behaviors that are demonstrated frequently in current practice as well as knowledge, skills, and behaviors that are less frequently evidenced in practice but presumed to become more prevalent in the future. Thus, the decision to use these documents reflects a limitation in the study. However, expectations for new graduates entering practice today are also in transition, as reflected by the regular revisions of outcome criteria prescribed by accreditation (1,2) and the movement of professional education toward the professional doctoral (DPT) degree, both of which obligate the profession to examine practice to keep pace with the changing expectations of employers, patients or clients, and payers.

The field studies do not represent an examination of the CPIs currently in use (the published forms are the fourth versions). The CPIs were modified in response to the results of the field studies. Examination of psychometric properties of the final versions of the CPIs remains to be done.

Fourth Drafts of the CPIs (Final Versions)

Modifications From the Previous Versions The fourth (final) versions of the CPIs were developed from July 1997 to September 1997 based on data from the field studies and feedback from physical therapy academic faculty, clinical educators, and students; consultation with the psychometrician; and changes in terminology found in the current versions of APTA documents, including the Guide, to Physical Therapist Practice, (17) A Normative Model of Physical Therapist Professional Education: Version `97, (26) and the first of 3 conferences on developing A Normative Model for Physical Therapist Assistant Education (performance expectations only) (Tabs. 12 and 13).

The final published versions of the instruments included the following: APTA disclaimer regarding the use of the instruments (ie, APTA disclaims any responsibility for controlling the manner in which any clinical education site may, based on the instruments, assess a student's clinical performance or an education institution may use them to determine a grading policy); copyright; table of contents; instructions for completion that included scoring and mechanisms for determining a grade, student, academic program, and clinical site information page; 24 PT and 20 PTA performance criteria; 5 "red-flag" performance criteria (items 1-5); summative comments for "areas of strengths," "needing improvement," and other comments; evaluation signatures; glossary; appendix of 3 examples of completed items for final and intermediate experiences; appendix of tests and measures and interventions (PT) and data collection and technical skills (PTA); appendix of performance criteria matched with Commission on Accreditation in Physical Therapy Education evaluative criteria for PT and PTA programs; historical perspective; and bibliography. Each performance criterion included the following components: performance criterion description, "Not Observed" boxes at midterm and final evaluations, VAS with 2 anchors ("Novice Clinical Performance" and "Entry-Level Performance"), "With Distinction" boxes for performance that exceeds expectations for the clinical experience at midterm and final evaluations, sample behaviors related to a level of performance that were modified for those items with high "Not Observed" responses on the field studies to include examples of behaviors that occur in common clinical settings, "Significant Concerns/At-Risk" boxes at midterm and final evaluations, midterm and final comments, and an explanation of qualitative dimensions that the task force believed apply in considering each rating. These 5 qualitative performance dimensions are quality of care, supervision/guidance required, consistency of performance, complexity of tasks/environment, and efficiency of performance (Figure, Appendix).

Instrument Adoption

In November 1997, the APTA Board of Directors approved the PT CPI for use as a voluntary instrument for assessing student clinical performance and made it available on a for-sale basis as a source of APTA non-dues revenue. The APTA Board of Directors approved the PTA CPI in March 1998 also as a for-sale item. Adoption of the PTA CPI was postponed in an effort to ensure that in the instrument there was congruence in language and performance expectations with A Normative Model for Physical Therapist Assistant Education: Version `98, (27) which had just been developed following the completion of 3 consensus-based conferences convened in the fall of 1997. Following a review of the consensus-based document, what the task force considered minor editorial changes in language within several performance expectations were made to the fourth version of the PTA CPI, again in an effort to ensure congruence with A Normative Model of Physical Therapist Assistant Education: Version `98. (27)

Future Research

The final published versions of the CPIs reflect input from clinical and academic communities, students, and a consultant psychometrician and analysis of the psychometric properties of 2 draft versions of the instruments. The task force believes this provides evidence to suggest that the CPIs in current use yield reliable and valid measurements of student clinical performance. Data, however, are lacking, and examination of the psychometric properties of the final published versions has not been done. This examination, the task force believes, should occur with ACCEs, CIs, and students who have been consistently trained in the use of the CPIs. The task force believes that psychometric properties to be examined should include reliability, validity, component structure, and users' satisfaction with the final published versions. Further studies of the CPIs might include the development of interpretive guidelines for their use as an evaluation leading to an academic grade; compilation of longitudinal data on the CPIs; comparison of the final published versions of the CPIs with computer-based versions of the CPIs (currently in development) for reliability, validity, user satisfaction, time required for completion, and ease of grading. In addition, longitudinal study of changes to frequently responded "Not Observed" performance criteria in relationship to changes occurring in education and practice is needed, and analysis of the use of the qualitative midterm and final comments and their relationship to the VAS score also is needed.

Conclusion

The PT CPI and the PTA CPI were developed using a multifaceted and sequential process. Data indicate that the task force considered acceptable psychometric properties for the first 3 drafts of the CPIs. The CPIs in use (fourth drafts), however, have not been studied, and users should consider this limitation.

Appendix.
Evolution of the Development of the Physical Therapist and Physical
Therapist Assistant Clinical Performance Instruments (CPIs) (a)

                                                     2nd Draft (Pilot
Features                 First Draft                 Study Version)

CPI disclaimer           Not included                Not included

Copyright declaration    Not included                Not included

Table of contents        Not included                Included

Instructions for         Directions included         Directions revised
  completion                                           for clarity

Student, site,           Not included                Not included
  program
  information page

Performance              23 PT/20 PTA criteria       23 PT/20 PTA
  criteria                                             revised criteria

"Red-flag"               "Red-flag"-identification   4 "red-flag"
    performance            of essential                 performance
    criteria               performance                  criteria
                           criteria                     identified

Sample behaviors         Sample behaviors            Sample behaviors
                           identified for each         revised for
                           performance                 performance
                           criterion                   criteria

Performance dimensions   Not included                Consider in rating
                                                       VAS quality,
                                                       consistency, and
                                                       guidance

VAS                      Left anchor "Novice         Left anchor
                           Student Clinician,"         "Novice Student
                           right anchor "Expert         Clinician,"
                           Clinician,"                  right anchor
                           midrange anchor              "Expert
                           slightly to the right        Clinician," and
                           of center "Entry-Level       mid-range
                           Clinician"                   anchor slightly
                                                        to the right of
                                                        center
                                                        "Entry-Level
                                                        Clinician"

"Not Observed" box      Not included                  Not included

"With Distinction" box  Not included                  Not included

"Significant Concerns"  Not included                  "Significant
  box                                                    Concerns/
                                                         At-Risk"
                                                         box-midterm
                                                         and final
                                                         placed at the
                                                         top of the
                                                         page

Performance criterion   Midterm and final             More space
  comments                comments for each             provided for
                          criterion                     midterm and
                                                        final comments

Summative comments      Not included                  Not included

Summative ratings       Included                      Comprehensive
                                                        ratings for
                                                        intermediate
                                                        and entry-level
                                                        experience

Evaluation signatures   Not included                  Midterm and final
                                                        student and
                                                        evaluator(s)
                                                        signatures

Glossary                Not included                  Included (denoted
                                                        by superscript
                                                        "g")

Appendices              Not included                   Not included

Historical              Preamble with                 Preamble included
perspective               historical                    historical
                          background,                   background,
                          rationale, and basic          rationale, and
                          assumptions at the            basic
                          front of the tool             assumptions at
                                                        the front of
                                                        the tool

Bibliography            Not included                  Included

Case vignette           Not included                  Case vignette
                                                        describing
                                                        "typical" new
                                                        graduate PT/PTA
                                                        at the front of
                                                        the tool

                        3rd Draft (Field Study       Final Published
Features                Version)                     Version

CPI disclaimer          Not included                 Included

Copyright declaration   Not included                 Included

Table of contents       Revised                      Revised

Instructions for        Directions reorganized and   Significant
  completion              revised for clarity          expansion

Student, site,          Included                     Included
  program
  information page

Performance             24 PT/20 PTA revised         2 PT/20 PTA
  criteria                criteria                     criteria-revised
                                                       based on
                                                       feedback and
                                                       results of field
                                                       study

"Red-flag"              5 "red-flag" performance        5 "red-flag"
    performance           criteria identified           performance
    criteria                                            criteria
                                                        identified
                                                        (items 1-5)

Sample behaviors        Sample behaviors revised     Sample behaviors
                          for performance              modified for
                          criteria based on            criteria based
                          pilot study                  on field study
                                                       and high "Not
                                                       Observed"
                                                       responses

Performance dimensions  Identified marking           Performance
                          definitions (ie,             dimensions
                          quality, supervision/        identified (ie,
                          guidance required,           quality of care,
                          consistency, complexity      supervision/
                          of tasks/environment,        guidance
                          and efficiency)              required,
                                                       consistency
                                                       of performance,
                                                       complexity of
                                                       tasks/
                                                       environment,
                                                       efficiency of
                                                       performance)
                                                       applicable to
                                                       all criteria

VAS                     Far left anchor "Novice      Far left anchor
                          Clinical Performance"        "Novice Clinical
                          and far right                Performance" and
                          anchor "Entry-Level          far right anchor
                          Performance"                 "Entry-Level
                                                        Performance"

"Not Observed" box      Included left of the VAS     Included left of
                                                       the VAS

"With Distinction" box  "With Distinction" box       "With Distinction"
                           signifies performance        box-performance
                           that exceeds entry           that exceeds
                           level                        expectations
                                                        for the
                                                        clinical
                                                        experience

"Significant Concerns"  Renamed "Significant         "Significant
  box                     Concerns" box on the          Concerns"
                          left side of page             box-midterm
                                                        and final
                                                        boxes in the
                                                        center of the
                                                        page with phone
                                                        icon

Performance criterion   Midterm and final            Midterm and final
  comments                comments for each            comments for
                          criterion                    each criterion

Summative comments      Areas of strength, areas     Areas of strength,
                          needing improvement,         areas needing
                          other comments               improvement,
                                                       other comments

Summative ratings       Summative ratings:           Removed summative
                          expectations for             ratings
                          clinical performance
                          and for entry level

Evaluation signatures   Midterm and final student    Midterm and final
                          and evaluator(s)             student and
                          signatures                   evaluator(s)
                                                       signatures

Glossary                New terms added to           Revised to be
                          glossary                     congruent with
                                                       APTA documents

Appendices              Three examples of a          Three examples of
                          completed item (PT and       completed items
                          PTA) located at the          (PT and PTA);
                          front of the tool            tests, measures,
                                                       and
                                                       interventions
                                                       (PT);
                                                       entry-level data
                                                       collection and
                                                       technical skills
                                                       (PTA);
                                                       Evaluative
                                                       Criteria for
                                                       Accreditation of
                                                       PT/PTA Programs
                                                       matched with CPI
                                                       performance
                                                       criteria

Historical perspective  Historical perspective,      Historical
                          preamble, rationale,         perspective
                          basic assumptions,           included
                          acknowledgments, and         preamble,
                          task force members,          rationale,
                          moved to the end of          assumptions,
                          the tool                     acknowledgments,
                                                       task force
                                                       members and
                                                       staff at the end

Bibliography            Included--enhanced           Included--enhanced

Case vignette           Case vignette revised and    Removed--see
                          moved to the end of the      Performance
                          tool                         Dimensions

(a) PT=physical therapist, PTA=physical therapist assistant,
VAS=visual analog scale, APTA=American Physical Therapy Association.
Table 1.
Second Draft (Pilot Study Version) of the Physical Therapist Clinical
Performance Instrument: Interrater Reliability of Scaled Items (a)

Item                                                       ICC
No.    Wording                                             (2,1)   N

 1.    Practices in a safe manner that minimizes risk to     .44   28
         patients, self, and others.
 2.    Demonstrates professional behavior during             .42   28
         interactions with others.
 3.    Adheres to ethical practice standards.                .38   28
 4.    Adheres to legal practice standards.                  .35   28
 5.    Communicates in ways that are congruent with          .42   28
         situational needs.
 6.    Produces documentation to support the delivery of     .42   28
         physical therapy services.
 7.    Incorporates cultural considerations into the         .11   28
         delivery of physical therapy care.
 8.    Applies principles of logic and scientific method     .60   28
         to the practice of physical therapy.
 9.    Screens patients using procedures to determine        .41   28
         the effectiveness of and need for physical
         therapy services.
10.    Performs a physical therapy examination.              .57   28
11.    Evaluates clinical findings to arrive at a            .53   28
         physical therapy diagnosis.
12.    Designs a physical therapy plan of care that          .59   28
         integrates goals, treatments, and discharge
         plan.
13.    Performs physical therapy treatment that achieves     .18   28
         desired patient outcomes.
14.    Educates others (patients, family, caregivers,        .38   28
         staff, students, other health care
         professionals) using relevant and effective
         teaching methods.
15.    Participates in activities assuring quality of        .21   28
         service delivery.
16.    Participates in physical therapy consultation         .62   28
         services.
17.    Address patient needs for services other than         .04   28
         physical therapy.
18.    Manages resources to achieve goals of the             .22   28
         practice setting.
19.    Participates in fiscal management of the physical    -.02   28
         therapy practice setting.
20.    Utilizes support personnel according to legal and     .18   28
         ethical guidelines.
21.    Recognizes that a PT has professional/social          .32   28
         responsibilities not definable in terms of work
         hours and job description.
22.    Formulates and implements a self-directed plan        .54   28
         for career development.
23.    Addresses prevention, wellness, and health            .14   28
         promotion needs of individuals, groups, and
         communities.
24.    Rate this student's overall performance relative      .59   28
         to academic and clinical expectations.
25.    Rate this student's overall performance relative      .17   28
         to entry level.

Novice Student
Clinician

Entry-Level
Clinician

Expert
Clinician

(a) PT=physical therapist, ICC:intraclass correlation coefficient,
N=number of students rated.
Table 2.
Second Draft (Pilot Study Version) of the Physical Therapist Assistant
Clinical Performance Instrument: Interrater Reliability of Scaled
Items (a)

Item                                                       ICC
No.    Wording                                             (2,1)   N

 1.    Practices in a safe manner that minimizes risk to     .70   10
           patients, self, and others.
 2.    Demonstrates professional behavior during             .74   10
         interactions with others.
 3.    Adheres to ethical practice standards.                .66   10
 4.    Adheres to legal practice standards.                  .54   10
 5.    Communicates in ways that are congruent with          .73   10
         situational needs.
 6.    Produces documentation to support the delivery of     .72   10
         physical therapy services.
 7.    Incorporates cultural considerations into the         .35   10
         delivery of physical therapy care.
 8.    Makes clinical decisions within the scope of PTA      .72   10
         practice.
 9.    Obtains accurate information by performing            .80   10
         selected tests and  measurements consistent
         with the plan of care.
10.    Participates in modifying the plan of care.           .90   10
11.    Performs physical therapy treatment based on a        .90   10
         plan of care established by a PT.
12.    Educates others (patients, family, caregivers,        .70   10
         staff, students, other health care
         professionals) using relevant and effective
         teaching methods.
13.    Participates in activities assuring quality of        .87   10
         service delivery.
14.    Participates in addressing patient needs for          .82   10
         services other than physical therapy.
15.    Manages resources to achieve goals of the             .68   10
         practice setting.
16.    Participates in fiscal management of the              .78   10
         physical therapy practice setting.
17.    Utilizes support personnel according to legal         .87   10
         and ethical guidelines.
18.    Recognizes that a PTA has responsibilities not        .70   10
         definable in terms of work hours and job
         description.
19.    Formulates and implements a self-directed plan        .68   10
         for career development.
20.    Assists the PT in addressing prevention,              .89   10
         wellness, and health promotion needs of
         individuals, groups, and communities.
21.    Rate this student's overall performance               .79   10
         relative to academic and clinical
         expectations.
22.    Rate this student's overall performance               .89   10
         relative to entry level.

Novice Student
Clinician

Entry-Level
Clinician

Expert
Clinician

(a) PT=physical therapist, PTA=physical therapist assistant,
ICC=intraclass correlation coefficient,
N=number of students rated.
Table 3.
Principal Component Analysis With Varimax Rotation of the Second Draft
(Pilot Study Version) of the Physical Therapist Clinical Performance
Instrument

                                                            Final
                                Component 1   Component 2   Communality
Item                            Loading       Loading       Estimate

 1. Practices in a safe
      manner.                   .39           .80           .80
 2. Professional behavior.      .37           .84           .85
 3. Ethical practice
      standards.                .34           .87           .89
 4. Legal practice standards.   .28           .85           .80
 5. Communicates.               .46           .82           .89
 6. Documentation.              .80           .43           .82
 7. Cultural considerations.    .41           .85           .90
 8. Logic and scientific
      method.                   .79           .40           .79
 9. Screens patients.           .75           .56           .88
10. Physical therapy
      examination.              .84           .43           .89
11. Physical therapy
      diagnosis.                .87           .42           .94
12. Physical therapy plan
      of care.                  .85           .42           .91
13. Physical therapy
      treatment.                .70           .62           .89
14. Educates others.            .85           .38           .87
15. Assuring quality of
      service delivery.         .87           .30           .85
16. Physical therapy
      consultation services.    .85           .33           .84
17. Need for other services.    .85           .30           .82
18. Manages resources.          .69           .61           .86
19. Fiscal management.          .86           .36           .88
20. Utilizes support
      personnel.                .79           .45           .84
21. Professional/social
      responsibilities.         .61           .66           .82
22. Career development.         .52           .73           .82
23. Prevention, wellness,
      and health promotion.     .78           .41           .78
24. Performance relative to
      expectations.             .19           .44           .23
25. Performance relative to
      entry level.              .72           .52           .79
Table 4.
Physical Therapist Clinical Performance Instrument Field Study:
Descriptive Statistics of the Final Score for Each Item

Item                                  % Not
No.    N     [bar]X   SD     Range    Observed

 1.    328   93.0     14.3   05-100   00.0
 2.    327   95.8     11.1   06-100   00.4
 3.    328   96.2     10.6   16-100   00.0
 4.    322   94.9     14.1   16-100   01.8
 5.    283   94.4     15.4   11-100   13.7
 6.    324   92.0     14.5   23-100   01.2
 7.    327   89.3     18.9   00-100   00.4
 8.    318   93.7     14.8   12-100   03.1
 9.    301   87.1     19.9   10-100   08.2
10.    258   87.7     21.5   04-100   21.4
11.    320   87.5     20.9   04-100   02.4
12.    315   85.3     22.2   05-100   04.0
13.    323   87.2     21.4   00-100   01.5
14.    322   88.8     20.1   03-100   01.8
15.    306   90.1     19.2   03-100   06.7
16.    225   86.7     20.5   00-100   31.4
17.    121   87.5     19.4   00-100   63.1
18.    201   85.2     20.2   04-100   38.7
19.    314   87.2     21.5   02-100   04.3
20.    262   90.1     17.6   05-100   20.1
21.    246   88.8     20.9   00-100   25.0
22.    285   90.6     19.2   00-100   13.1
23.    317   91.0     18.9   09-100   03.4
24.    144   85.5     22.1   05-100   56.1
25.    313   97.4     07.7   50-100   04.6
26.    298   89.1     19.6   06-100   14.0
Table 5.
Physical Therapist Assistant Clinical Performance Instrument Field
Study: Descriptive Statistics of the Final Score for Each Item

Item                                  % Not
No.    N     [bar]X   SD     Range    Observed

 1.    192   90.9     14.7   18-100   00.5
 2.    191   94.1     11.6   29-100   01.0
 3.    192   94.0     11.5   15-100   00.5
 4.    190   95.1     11.2   06-100   01.5
 5.    190   94.0     12.5   34-100   01.5
 6.    193   89.7     15.3   07-100   00.0
 7.    189   87.8     17.1   03-100   02.1
 8.    184   93.7     13.5   03-100   04.7
 9.    182   85.9     18.5   00-100   05.7
10.    183   86.7     19.1   00-100   05.2
11.    181   85.2     19.6   04-100   06.2
12.    189   88.2     16.8   07-100   02.1
13.    173   85.2     19.8   00-100   10.4
14.    150   82.8     22.2   01-100   22.3
15.    116   81.4     22.8   01-100   39.9
16.    184   87.1     18.8   08-100   04.7
17.    139   88.6     18.4   03-100   28.0
18.    146   84.7     21.1   17-100   24.3
19.    183   89.9     15.5   23-100   05.2
20.     87   82.7     23.8   05-100   54.9
21.    188   95.3     09.6   40-100   02.6
22.    170   90.3     16.7   00-100   11.9
Table 6.
Characteristics of Physical Therapist (PT) and Physical Therapist
Assistant (PTA) Students Participating in the Clinical Performance
Instrument Field Studies

                                             PTA        PT
                                             Students   Students
                                             (n=181)    (n=319)

Percentage male                              31.1       32.0
Percentage on first clinical experience      37.6       12.0
Percentage on final clinical experience      11.6       10.4
Age (y)
  [bar]X                                     29.0       26.7
  SD                                          7.5        5.5
  Range                                      19-51      20-47

Length of current clinical experience (wk)
  [bar]X                                      5.1        6.5
  SD                                          2.0        3.6
  Range                                       2-16       2-51

Months of course work prior to clinical
  experience
  [bar]X                                     10.9       16.4
  SD                                          7.3        7.1
  Range                                       0-34       1-45

Total days of clinical experience
  [bar]X                                     68.5       71.1
  SD                                         39.8       45.1
  Range                                      10-272     10-363
Table 7.
Third Draft (Field Study Version) of the Physical Therapist Clinical
Performance Instrument: Interrater Reliability of Scaled Items

                                                         ICC
Item   Wording                                           (2,1)   N

 1.    Practices in a safe manner that minimizes risk
         to patients, self, and others.                  .44    30

 2.    Presents self in a professional manner.           .57    30

 3.    Demonstrates professional behavior during
         interactions with others.                       .52    30

 4.    Adheres to ethical practice standards.            .58    30

 5.    Adheres to legal practice standards.              .35    19

 6.    Communicates in ways that are congruent
         with situational needs.                         .57    29

 7.    Produces documentation to support the
         delivery of physical therapy services.          .62    29

 8.    Adapts delivery of physical therapy care
         to reflect respect for and sensitivity
         to individual differences                       .57    28

 9.    Applies principles of logic and
         scientific method to the practice of
         physical therapy.                               .64    23

10.    Screens patients using procedures to
         determine the effectiveness of and
         need for physical therapy services.             .41    17

11.    Performs a physical therapy examination.          .52    25

12.    Evaluates clinical findings to arrive at
         a physical therapy diagnosis.                   .64    25

13.    Designs a physical therapy plan of care that
         integrates goals, treatments, and
         discharge plan.                                 .61    25

14.    Performs physical therapy interventions in a
         technically competent manner.                   .46    27

15.    Educates others (patients, family, caregivers,
         staff, students, other health care
         professionals) using relevant and effective
         teaching methods.                               .70    25

16.    Participates in activities assuring quality
         of service delivery.                            .38    16

17.    Provides consultation to individuals and
         outside organizations.                                  1

18.    Addresses patient needs for services other
         than physical therapy.                          .76    10

19.    Manages resources to achieve goals of the
         practice setting.                               .67    27

20.    Incorporates an understanding of economic
         factors in the delivery of physical
         therapy services.                               .32    15

21.    Utilizes support personnel according to legal
         and ethical guidelines.                         .40    19

22.    Recognizes that a PT has professional/social
         responsibilities not definable in terms of
         work hours and job description.                 .46    19

23.    Implements a self-directed plan for
         professional development and lifelong
         learning.                                       .48    28

24.    Addresses prevention, wellness, and health
         promotion needs of individuals, groups,
         and communities.                                .70     7

25.    Rate this student's overall performance
         relative to academic and clinical
         expectations.                                   .21    28

26.    Rate this student's overall performance
         relative to entry level.                        .29    25

Novice Clinical
Performance

Entry-Level
Performance

(a) PT=physical therapist, ICC=intraclass correlation coefficient,
N=number of students rated. Boldface type indicates that the wording
of the item was changed from the previous version.
Table 8.
Third Draft (Field Study Version) of the Physical Therapist
Assistant Clinical Performance Instrument: Interrater Reliability of
Scaled Items (a)

Item                                                     ICC
No.    Wording                                           (2,1)   N

 1.    Practices in a safe manner that minimizes
         risk to patients, self, and'others.              .56    20

 2.    Conducts self in a responsible manner.             .16    20

 3.    Interacts with others in a professional manner.    .13    21

 4.    Adheres to ethical practice standards.            -.12    20

 5.    Adheres to legal practice standards.               .01    19

 6.    Communicates in ways that are congruent with
         situational needs.                               .00    20

 7.    Produces documentation to support the delivery
         of physical therapy services.                    .01    19

 8.    Adapts delivery of physical therapy care to
         reflect respect for and sensitivity to           .30    19
         individual differences.

 9.    Makes decisions within the clinical
         environment based on an established physical
         therapy plan of care.                            .10    20

10.    Obtains accurate information by performing
          selected tests and measurements consistent
          with the plan of care.                          .61    18

11.    Participates in modifying the plan of care.        .49    19

12.    Performs physical therapy interventions in a
         technically competent manner.                    .13    20

13.    Educates others (patients, family, caregivers,
         staff, students, other health care
         professionals) using relevant and
         effective teaching methods.                      .00    17

14.    Participates in activities addressing quality
         of service delivery.                             .25    16

15.    Participates in addressing patient needs for
         services other than physical therapy.            .21     4

16.    Manages resources to achieve goals of the
         practice setting.                               -.12    18

17.    Participates in fiscal management of the
         physical therapy practice setting.               .21    13

18.    Utilizes PT aides and other support personnel
         according to legal and ethical guidelines.       .23    11

19.    Formulates and implements a self-directed plan
         for career development.                          .31    18

20.    Assists the PT in addressing prevention,
         wellness, and health promotion needs of
         individuals, groups, and communities.            .13     5

21.    Rate this student's overall performance
         relative to academic and clinical
         expectations.                                    .78    17

22.    Rate this student's overall performance
         relative to entry level.                         .67    13

Novice Clinical
Performance

Entry-Level
Performance

(a) PT=physical therapist, ICC=intraclass correlation coefficient,
N=number of students rated. Boldface type indicates that the wording
of the item was changed from the previous version.
Table 9.
Field Study: Construct Validity of Third Draft of the Physical
Therapist Clinical Performance Instrument (a)

Item   First vs Last Clinical Experience (a)
No.    Mean First   Mean Last                  P

 1.    80.7         99.1                       .0001
 2.    89.0         96.3                       .0853
 3.    87.5         98.9                       .0028
 4.    83.1         99.8                       .0006
 5.    80.6         99.9                       .0016
 6.    80.9         98.1                       .0001
 7.    67.7         98.2                       .0001
 8.    79.6         98.8                       .0004
 9.    68.8         96.1                       .0001
10.    72.9         96.5                       .0001
11.    69.5         96.3                       .0003
12.    65.2         97.1                       .0001
13.    69.5         97.8                       .0001
14.    71.4         98.6                       .0001
15.    72.8         97.7                       .0002
16.    72.2         97.0                       .0013
17.    78.9         98.0                       .0348
18.    68.5         94.1                       .0112
19.    68.8         97.6                       .0001
20.    82.4         96.1                       .0336
21.    79.6         98.0                       .0142
22.    73.7         98.1                       .0008
23.    74.0         98.5                       .0001
24.    65.9         99.5                       .0187
25.    96.0         98.8                       .1071
26.    70.6         98.0                       .0001

       Days of Clinical
Item   Experience (b)             Social Skills Inventory (b)
No.    Correlation        P       Correlation   P

 1.    .33                .0001    .04          .7987
 2.    .22                .0001    .16          .3963
 3.    .27                .0001    .02          .8800
 4.    .31                .0001    .01          .9201
 5.    .30                .0001    .25          .2211
 6.    .33                .0001    .04          .8190
 7.    .40                .0001   -.02          .8800
 8.    .34                .0001    .15          .4100
 9.    .38                .0001   -.11          .5767
10.    .35                .0001    .10          .6926
11.    .38                .0001    .12          .5379
12.    .36                .0001   -.06          .7580
13.    .38                .0001   -.21          .2693
14.    .40                .0001    .01          .9246
15.    .34                .0001    .00          .9938
16.    .34                .0001    .05          .8283
17.    .30                .0001    .08          .7859
18.    .32                .0001   -.05          .8392
19.    .36                .0001    .12          .5075
20.    .27                .0001    .00          .9914
21.    .31                .0001    .03          .8936
22.    .33                .0001    .15          .4341
23.    .34                .0001    .13          .4925
24.    .29                .0005    .05          .8610
25.    .12                .0317   -.06          .7389
26.    .40                .0001    .07          .7025

(a) Student t test.

(b) Pearson correlation.
Table 10.
Field Study: Construct Validity of the Third Draft of the
Physical Therapist Assistant Clinical Performance Instrument

Item   First vs Last Clinical Experience (a)
No.    Mean First   Mean Last                  P

 1.    85.2         98.9                       .0001
 2.    91.3         99.4                       .0001
 3.    89.9         98.7                       .0001
 4.    90.6         99.8                       .0001
 5.    89.1         99.5                       .0001
 6.    82.5         98.3                       .0001
 7.    79.4         98.9                       .0001
 8.    89.1         95.5                       .0001
 9.    77.8         97.8                       .0001
10.    77.8         98.8                       .0001
11.    74.9         98.5                       .0001
12.    79.9         98.7                       .0001
13.    77.2         99.1                       .0001
14.    77.5         98.1                       .0001
15.    70.4         96.9                       .0001
16.    78.2         98.3                       .0001
17.    80.3         98.9                       .0001
18.    75.9         99.5                       .0001
19.    85.0         98.2                       .0001
20.    73.0         98.1                       .0001
21.    93.2         99.9                       .0001
22.    82.8         99.9                       .0001

        Days of Clinical
Item    Experience (b)              Social Skills Inventory (b)
No.     Correlation         P       Correlation   P

 1.     .27                 .0002    .41          .0113
 2.     .14                 .0588    .18          .2788
 3.     .20                 .0061    .19          .2372
 4.     .14                 .0528    .14          .3967
 5.     .25                 .0012    .00          .9697
 6.     .23                 .0022    .26          .1043
 7.     .31                 .0001    .11          .4834
 8.     .22                 .0032    .18          .2856
 9.     .27                 .0004    .19          .2648
10.     .27                 .0004    .18          .2649
11.     .34                 .0001    .09          .5844
12.     .32                 .0001    .04          .7784
13.     .31                 .0001    .09          .6031
14.     .32                 .0001    .37          .0765
15.     .32                 .0008    .04          .8131
16.     .22                 .0039    .21          .2833
17.     .37                 .0001    .15          .4252
18.     .33                 .0001    .05          .7540
19.     .17                 .0253    .05          .7540
20.     .26                 .0169    .42          .1515
21.     .12                 .1000    .05          .7390
22.     .22                 .0054   -.02          .8860

(a) Student t test.

(b) Pearson correlation.
Table 11.
Satisfaction with the Third Draft (Field Study Version) of the
Clinical Performance Instrument (a)

                           PT       PTA
                           Student  Student  PT CI   PTA CI  ACCE
How satisfied with:        Median   Median   Median  Median  Median

Clarity of instructions
  comments?                5        5        6       5       5
Clarity of instructions
  for VAS?                 5        5        6       5       5
Value of using "red-flag"
  criteria?                5        5        6       5       6
Usefulness of the VAS?     5        4        5       5       5
Ability to distinguish
  pass/fail?               5        5        5       5       5
As a basis for feedback?   5        5        5       5       6
As a Basis for grading?    5        4        4       4       3
Time to complete?          4        4        4       4       5
Ease of use?               5        4        5       5       5
Comprehensiveness?         5        5        5       5       5
Identifying a problem?     5        5        5       5       6
Identifying nature of a
  problem?                 5        5        5       5       5
Developing a plan to
  address problem?         5        4        5       4       5
Identifying problems
  with technical
  skills?                  4        4        5       4       4
Identifying weakness
 in academic curriculum?   4        4        4       4       4
Usefulness of performance
  criteria examples?       5        5        5       5       6
Compared with other
  instruments, how would
  you rate this one?       NA       NA       5       4       5

(a) PT=physical therapist, PTA=physical therapist assistant,
CI=clinical instructor, ACCE=academic coordinator of physical therapy,
VAS=visual analog scale,

NA=not applicable.

Table 12.

Fourth Draft (Final Version) of the Physical Therapist Clinical Performance Instrument: Items and Measurement Scale"

1. Practices in a safe manner that minimizes risk to patients, self, and others.

2. Presents self in a professional manner.

3. Demonstrates professional behavior during interactions with others.

4. Adheres to ethical practice standards.

5. Adheres to legal practice standards.

6. Communicates in ways that are congruent with situational needs.

7. Produces documentation to support the delivery of physical therapy services.

8. Adapts delivery of physical therapy care to reflect respect for and sensitivity to individual differences.

9. Applies principles of logic and scientific method to the practice of physical therapy.

10. Screens patients using procedures to determine the effectiveness of and need for physical therapy services.

11. Performs a physical therapy examination.

12. Evaluates clinical findings to arrive at a physical therapy diagnosis and outcomes of care.

13. Designs a physical therapy plan of care that integrates goals, treatment, outcomes, and discharge plan.

14. Performs physical therapy interventions in a technically competent manner.

15. Educates others (patients, family, caregivers, staff, students, other health care providers) using relevant and effective teaching methods.

16. Participates in activities assuring quality of service delivery.

17. Provides consultation to individuals, businesses, schools, government agencies, or other organizations.

18. Addresses patient needs far services other than physical therapy as needed.

19. Manages resources (eg, time, space, equipment) to achieve goals of the practice setting.

20. Incorporates an understanding of economic factors in the delivery of physical therapy services.

21. Utilizes support personnel according to legal and ethical guidelines.

22. Demonstrates that a PT has professional/social responsibilities beyond those defined by work expectations and lob description.

23. Implements a self-directed plan for professional development and lifelong learning.

24. Addresses primary and secondary prevention, wellness, and health promotion needs of individuals, groups, and communities.

Novice Clinical Performance

Entry-Level Performance

(a) PT=physical therapist. Boldface type indicates that the wording of the item was changed from the previous version.

Table 13.

Fourth Draft (Final Version) of the Physical Therapist Assistant Clinical Performance Instrument: Items and Measurement Scale (a)

1. Performs in a safe manner that minimizes risk to patients, self, and others.

2. Conducts self in a responsible manner.

3. Interacts with others in a respectful manner.

4. Adheres to ethical practice standards.

5. Adheres to legal standards.

6. Communicates in ways that are congruent with situational needs.

7. Produces documentation to support the delivery of physical therapy services.

8. Delivers established patient care to reflect respect far and sensitivity to individual differences.

9. Participates in patient status judgments within the clinical environment based on the plan of care established by the PT.

10. Obtains accurate information by performing selected data collection consistent with the plan of care established by the PT.

11. Discusses need for modifications to the plan of care established by the PT.

12. Performs physical therapy interventions in a technically competent manner.

13. Educates others (patients, family, caregivers, staff, students, other health care providers) using relevant and effective teaching methods.

14. Participates in activities addressing quality of service delivery.

15. Participates in addressing patient needs for services other than physical therapy.

16. Manages resources (eg, time, space, and equipment) to achieve goals of the clinical setting.

17. Participates in fiscal management of the physical therapy practice setting.

18. Utilizes PT aides and other support personnel according to legal standards and ethical guidelines.

19. Implements a self-directed plan for career development and lifelong learning.

20. Assists the PT in addressing primary and secondary prevention needs of individuals and groups.

Novice Clinical Performance

Entry-Level Performance

(a) PT=physical therapist. Boldface type indicates that the wording of the item was changed from the previous version.

Funding from the American Physical Therapy Association supported the work of Task Force for the Development of Student Clinical Performance Instruments.

References

(1) Commission on Accreditation in Physical Therapy Education. Evaluative Criteria for Accreditation of Education Programs for the Preparation of Physical Therapists. In: Accreditation Handbook. Alexandria, Va: American Physical Therapy Association; October 30, 1996.

(2) Commission on Accreditation in Physical Therapy Education. Evaluative Criteria for Accreditation of Education Programs for the Preparation of Physical Therapist Assistants. In: Accreditation Handbook. Alexandria, Va: American Physical Therapy Association; July 1, 1994.

(3) Kassebaum DG. The measurement of outcomes in the assessment of educational program effectiveness. Acad Med. 1990;65:293-296.

(4) Nethery JC. Physical therapy competencies in clinical education. Phys Ther. 1981;61:1442-1446.

(5) May BJ. Competency-based evaluation of student performance. J Allied Health. 1978;7:232-237.

(6) Teschendorf B, Gramet P, Heubusch L. Group development of a clinical education instrument. Journal of Physical Therapy Education. 1988;2(1):10-12. (New York Consortium of Academic Coordinators. New York State Physical Therapy Student Performance Evaluation, 1992.)

(7) The Blue MACS: Mastery and Assessment of Clinical Skills. 4th ed. Galveston, Tex: Texas Consortium for Physical Therapy Education; 1989.

(8) Hrachovy J, Clopton N, Baggett K, et al. Use of the Blue MACS: acceptance by clinical instructors and self-reports of adherence. Phys Ther. 2000;80:652-661.

(9) Dickinson R, DiMarino J, Pfitzenmaier J. A common evaluation instrument. Phys Ther. 1973;53:1075-1080.

(10) May WW, Morgan BJ, Lemke JC, et al. Development of a model for ability-based assessment in physical therapy education: one program's experience. Journal of Physical Therapy Education. 1995;9(1):3-6.

(11) Lewthwaite R. Development and Use of the Clinical Competence Scale (CCS) [unpublished manuscript]. Downey, Calif: Rancho Los Amigos Medical Center; September 1994.

(12) Loomis J. Evaluating clinical competency of physical therapy students, part 2: assessing the reliability, validity, and usability of a new instrument. Physiotherapy Canada. 1985;37:91-98.

(13) APTA Clinical Education Resource Guide: Summary of Clinical Education Evaluation Tools. Alexandria, Va: American Physical Therapy Association; 1995.

(14) Program 60, Education Division, Exhibit 45; Final Report: Task Force on Clinical Education (1992-1994); American Physical Therapy Association Board of Directors, November 1994.

(15) Messick S. The interplay of evidence and consequences in the validation of performance assessments. Educ Res. 1994;23:13-23.

(16) Friedman M, Mennin SP. Rethinking critical issues in performance assessment. Acad Med. 1991;66:390-395.

(17) Guide to Physical Therapist Practice, Part One: A Description of Patient/Client Management. Phys Ther. 1995;75:707-764.

(18) Cox K. Examining and recording clinical performance: a critique and some recommendations. Education for Health: Change in Training and Practice. 2000;13:45-52.

(19) Wewers ME, Lowe NK. A critical review of visual analog scales in the measurement of clinical phenomena. Res Nurs Health. 1990;13: 227-236.

(20) Ohnhaus EE, Adler R. Methodological problems in the measurements of pain: a comparison between the verbal rating and the visual analog scale. Pain. 1975;1:379-384.

(21) Streiner DL, Norman GR, eds. Health Measurement Scales: A Practical Guide to Their Development and Use. 2nd ed. Oxford, England: Oxford University Press; 1995.

(22) SAS/STAT User's Guide, Version 6, Volumes 1 and 2. 4th ed. Cary, NC: SAS Institute Inc; 1989.

(23) Portney LG, Watkins MP, eds. Foundations of Clinical Research: Applications to Practice. East Norwalk, Conn: Appleton & Lange; 1993.

(24) Anastasi A. Psychological Testing. 3rd ed. New York, NY: Macmillan Publishing Co Inc; 1968.

(25) Woehr DJ. Rater training for performance appraisal: a quantitative review. Journal of Occupational and Organizational Psychology. 1994;67: 189-205.

(26) A Normative Model of Physical Therapist Professional Education (Version `97). Alexandria, Va: American Physical Therapy Association; 1997.

(27) A Normative Model of Physical Therapist Assistant Education (Version `98). Alexandria, Va: American Physical Therapy Association; 1998.

(28) Riggio RE. Social Skills Inventory. Palo Alto, Calif: Consulting Psychologists Press; 1989.

(29) Riggio RE. Assessment of basic social skills. J Pers Soc Psych. 1986;51: 649-660.

(30) Rothstein JM, Echternach JL. Primer on Measurement: An Introductory Guide to Measurement Issues. Alexandria, Va: American Physical Therapy Association; 1993.

Members of the American Physical Therapy Association's Task Force for the Development of Student Clinical Performance Instruments:

Kathryn Roach, PT, PhD, is Associate Professor, Department of Physical Therapy and Department of Orthopaedics and Rehabilitation, and Associate Director for Research, University of Miami School of Medicine, Coral Gables, Fla. She was Assistant Professor, Department of Physical Therapy and Department of Orthopaedics and Rehabilitation, University of Miami School of Medicine, at the time of the study. She was primarily involved in the conceptual development, design, ongoing revision of the instruments, design of pilot and field studies, management and coordination of data entry with the assistance of graduate students from the University of Miami, and resultant data analysis. She wrote portions of this article and reviewed and edited the final manuscript prior to submission.

Jody Gandy, PT, PhD, is Director, Department of Physical Therapy Education, American Physical Therapy Association, Alexandria, Va. She was Director of Clinical Education, American Physical Therapy Association, at the time of the study. She served as staff liaison to this group and was responsible for project management by ensuring that the Task Force met its charge in accordance with American Physical Therapy Association policies and expected timelines, budgeted and procured the necessary funds to complete the project, and assumed responsibility for data collection. She wrote portions of this article and reviewed, coordinated, and edited the final manuscript prior to submission.

Susan S Deusinger, PT, PhD, is Director, Program in Physical Therapy, and Associate Professor of Neurology, Washington University School of Medicine, St Louis, Mo. She was primarily involved in the conceptual development, design, ongoing revisions, design of pilot and field studies, and resultant data analysis, and she reviewed and edited the final manuscript prior to submission.

Sherry Clark, PT, MS, is Associate Professor, School of Pharmacy and Allied Health Professions, Creighton University, Omaha, Neb. She was Center Coordinator of Clinical Education, Shepherd Spinal Center, Department of Physical Therapy, Atlanta, Ga, and Associate Professor and Co-Academic Coordinator of Clinical Education, Department of Physical Therapy, School of Allied Health Sciences and Graduate Studies, Medical College of Georgia, Augusta, Ga, at the time of the study. She served as co-chair of this group and was primarily involved in the conceptual development, design, and ongoing revision of the instruments.

Pamela Gramet, PT, PhD, is Chairperson and Associate Director, Department of Physical Therapy Education, State University of New York Upstate Medical University, Syracuse, NY. She was Associate Professor and Academic Coordinator of Clinical Education, Department of Physical Therapy Education, College of Health Profession, SUNY Health Sciences Center at Syracuse, State University of New York at Syracuse, Syracuse, NY, at the time of the study. She was primarily involved in the conceptual development, design, and ongoing revision of the instruments.

Barbara Gresham, PT, MS, is Instructor, Physical Therapist Assistant Program, McLennan Community College, Waco, Tex. She was Program Director, Physical Therapist Assistant Program, McLennan Community College, Waco, Tex, at the time of the study. She served as co-chair of this group. She was primarily involved in the conceptual development, design, ongoing revision of the instruments, and review of the manuscript before submission.

Paul Hagler, PhD, is Associate Dean, Graduate Studies and Research Faulty of Rehabilitation Medicine, University of Alberta, Edmonton, Canada. He was Professor and Director, Faculty of Rehabilitation Medicine, Centre for Studies in Clinical Education, University of Alberta, at the time of the study. He was primarily involved in the conceptual development, design, ongoing revision of the instruments, design of pilot and field studies, resultant data analysis, and review of the manuscript before submission.

Rebecca Lewthwaite, PhD, is Director of Research and Education in Physical Therapy, Rancho Los Amigos National Rehabilitation Center, Rancho Los Amigos Medical Center, Downey, Calif. She was Director, Center for Research in Biokinesiology, and Associate Director of Physical Therapy for Research and Education, Rancho Los Amigos Medical Center, at the time of the study. She was involved in the conceptual development, design, ongoing revision of the instruments, design of pilot and field studies, and resultant data analysis.

Bella J May, PT, EdD, FAPTA, is President, BJM Enterprises, PC, Dublin, Calif, and Professor Emerita, Medical College of Georgia, Augusta, Ga. She was Professor and Co-Academic Coordinator of Clinical Education, Department of Physical Therapy, School of Allied Health Sciences and Graduate Studies, Medical College of Georgia, at the time of the study. She was primarily involved in the conceptual development, design, and ongoing revision of the instruments.

Babette Sanders, PT, MS, is Assistant Professor, Department of Physical Therapy and Human Movement Science, Northwestern University Medical School, Chicago, Ill. She was Instructor, Programs in Physical Therapy, Northwestern University Medical School, and Part-time Clinician, Department of Physical Therapy, Evanston Hospital, Evanston, Ill, at the time of the study. She was primarily involved in the conceptual development, design, and ongoing revision of the instruments.

Michael J Strube, PhD, is Professor, Department of Psychology, Washington University, St Louis, Mo. He served as consultant psychometrician from July 1996 to April 1997. He provided assistance in field study design and construction, data analysis, enhancement of psychometric properties of the instruments using a visual analog scale, and he reviewed and commented on the final manuscript before submission.

Yolanda Rainey, PT, MS, is Area Supervisor, Rehab Outreach of St Louis, Mo, at Oakwood Nursing and Rehabilitation Center, Virginia Beach, Va. She was Director of Rehabilitation and Center Coordinator of Clinical Education, NovaCare Geriatric Center, Portsmouth, Va, at the time of the study. She served on this group from 1994 to 1995 and was primarily involved in the initial development of the first draft versions of the instruments.