Printer Friendly

Evaluating clinical problem-solving skills through computer simulations.

Evaluating Clinical Problem-Solving Skills Through Computer Simulations

Computer-aided instruction (CAI) is an effective teaching technique that has existed for almost two decades (Braun, 1980; Frenzel, 1980; Gerhold, 1978; Office of Technology Assessment, 1979). The primary advantage of CAI over traditional classroom teaching methods appears to be the automatic interaction and immediate feedback that the computer can provide to students on an individual basis (Frenzel, 1980). In addition, CAI is an exciting, interesting, and valid way of introducing students to the use of modern computers. Unfortunately, because of the historically high cost of electronic computing on large mainframe computers and medium-size minicomputers, CAI has not been used extensively. However, with the advent of large-scale integrated (LSI) circuits and the introduction of personal microcomputers, the cost of CAI computing is no longer prohibitive. As a result, there is currently a resurgent interest in the application of computers in education (Frenzel, 1980).

In rehabilitation, the provision of high-quality services to large numbers of clients with a wide variety of disabling conditions and rehabilitation needs is a great challenge to counselors (Cox, Connolly, & Flynn, 1981). To meet this challenge, rehabilitation counselors must develop not only strong clinical and counseling skills, but must also possess strong case management skills in the areas of problem-solving, time usage, budgeting, communication, use of community resources, and documentation as well. Although the professional preparation of rehabilitation counselors devotes a considerable amount of time to developing clinical and counseling skills, it has been only recently that consideration has been given to preparation in case management (Cox, Connolly, & Flynn, 1981). This lack of attention can be attributed to the fact that case management skills are more readily developed through hands-on experience and it is difficult to simulate this experience in a classroom setting.

To facilitate training in this area, computerized case management simulation programs have been developed to help rehabilitation counselor trainees develop appropriate clinical problem-solving and case management skills. Because undergraduate and graduate level rehabilitation services students are frequently employed as rehabilitation counselors in state vocational rehabilitation agencies, it is logical to assume that they could also benefit from case simulation exercises. Berven and his colleagues have developed a series of case management simulation programs (based on state vocational rehabilitation agency materials) on large mainframe computers (Berven, 1985; Berven & Scofield, 1980) and microcomputers (Chan, Parker, Lam, Mecaskey & Malphurs, 1987). They used computer simulations to compare clinical and graduate problem-solving skills of undergraduate and graduate rehabilitation students with experienced rehabilitation counselors. The computer simulation results have provided encouraging reliability and validity data and have successfully differentiated case performance among rehabilitation counseling students (Berven, 1985; Berven & Scofield, 1980; Chan et al., 1987). However, Berven and Scofield's original 1980 study, with only 12 counselors forming the performance standard for comparison purposes is tentative at best. If a study with a large representative sample of counselors and students is not possible, then a series of local validation studies conducted across the country maybe necessary to provide further evidence of psychometric properties for computer simulations to be considered a valid assessment and teaching tool in rehabilitation education.

The purpose of this study was to validate the appropriateness of the computerized case simulation program developed by Berven and Scofield (1980) as a tool to evaluate case management performance using a sample of rehabilitation students and rehabilitation counselors in the state of Texas.



Participants were 33 undergraduate rehabilitation services students and 13 graduate rehabilitation counseling students from three major universities in Texas, and 19 Texas Rehabilitation Commission (TRC) counselors. The mean age of participants was 37.32 years (SD = 10.87), and 58.4% (n = 38) were female. The student participants had varying degrees of clinical experience, ranging from first semester practicum students (51.1%), second semester practicum students (33.3%) to those who were in the third semester of their clinical experience (15.5%). The minimal clinical experience requirement for participation in this research was similar to Berven and Scofield's (1980) original study of graduate rehabilitation counseling trainees. The TRC counselors had a minimum of a bachelor's degree, and had worked for TRC for at least 6 months. All participants were volunteers.

Computerized Case-Management Simulation

The original computerized case simulation program was based on case simulation exercises that had been developed by rehabilitation researchers at the University of Wisconsin-Stout, using actual state VR agency case files (Berven & Scofield, 1980). The case materials were computerized by Berven and Scofield (1980) by means of a BASIC program written for the UNIVAC 1110 mainframe computer (Berven & Scofield, 1980). In the present study, the IBM PC version of the case management simulation program developed by Chan et al. (1987) was used.

The case used for this simulation involved a 20-year-old male with a seizure disorder that was a result of a head injury. The client was self-referred. In completing the simulation, the participants assumed the role of a state vocational rehabilitation counselor charged with determining the course of actions to be taken with the client. In addition to the computer simulation, the participants were provided with oral instructions on how to operate the microcomputer and general instructions about the task to be performed. The instructions were that the client had come to the state agency when the counselor was unavailable, had filled out an application form, and then left. The participants entered the simulation at this point.

When the participants felt that enough information had been gathered, they were asked a series of questions by the computer. These questions were: 1) What is the primary disability of this client, if any? 2) What is the secondary disability, if any? 3) Does the disability (or disabilities) result in a handicap to employment? 4) Is the client eligible for services? 5) Will services enable the client to be employable? 6) What are the chances that rehabilitation services will result in the client's becoming employed?

As a means of arriving at these decisions, the simulation provided a total of 21 possible actions. These included: 10 medical and specialist reports; a psychological report; a work verification; a social services report; education history; a 14-day vocational evaluation; and a maximum of 6 interviews with the client. When interviews were requested, they were presented in a predetermined order, and summarized information obtained directly from the client.

Reports from the various services could be requested one at a time. A list of possible actions was then presented on the monochrome display. After reading information from an action taken, the participant could then request another action from those still available. This sequence was then repeated until the participant felt that sufficient information had been obtained to make the determinations. At this point, the computer presented the series of questions regarding each determination, and the participants typed in their responses. The simulation required approximately 30 minutes to complete.


The participants were given verbal instructions as to the purpose of the simulation and the operation of the computer. Once they indicated that they understood the purpose of the research, they were asked to proceed with the computer simulation using an IBM PC. The researchers observed the participants for the first five minutes to provide assistance if difficulties arose. Then, they were left alone to finish the simulation. The computer simulation results for each participant were automatically stored on a floppy disk by the IBM PC.

Data Analysis

The frequency counts and proportions of the sample of students or counselors taking each action in completing the simulation were tabulated and compared with the results obtained in Berven and Scofield's study. In addition, a complete-link hierarchical cluster analysis (Johnson, 1967) was performed to separate students into homogeneous subgroups in terms of their problem-solving approaches. Performance of these undergraduate and graduate rehabilitation students was compared to performance of experienced rehabilitation counselors derived from the group of the Texas Rehabilitation Counselors as well as the original counselors from Berven and Scofield's study. Furthermore, to better explain action pattern differences of these student groups, a step-wise discriminant analysis was conducted using membership in student clusters as the criterion variable and the 21 actions as predictors.


The frequency counts and proportions of the sample of students taking each action in completing the simulation were tabulated. These results along with the results for TRC counselors (our local criterion group) and Berven and Scofield's counselors (the original criterion group) are presented in Table 1.

As can be observed in Table 1, TRC counselors in our study selected the following actions more frequently than undergraduate students: 1) the general medical examination (94.7% vs. 72.7%); 2) neurological examination (84.2% vs. 66.7%); 3) psychological examination (89.5% vs. 69.7%); 40 Interview 2 (78.9% vs 60.6%); Interview 3 (78.9% vs 54.5%); 5) Interview 4 (73.7% vs 45.5%); 6) Interview 5 (68.4% vs 36.4%); and 6) Interview 6 (52.6% vs 33.3%). General medical, neurological, and psychological evaluations are all considered appropriate action in working with this simulated client. More students in our study than TRC counselors were inclined to choose the educational history (33.3% vs. 10.5%). Seven special medical examinations (cardiovascular, gastrointestinal, orthopedic, respiratory, speech and hearing, urological, and visual examinations) were considered unnecessary by all TRC counselors in the criterion group, but were selected by some of the students in our study.

When comparing our graduate students to TRC counselors, they appeared to make choices more similar to the counselors than did the undergraduates. Of the actions considered unnecessary by the counselors, the graduate students considered two actions (cardiovascular and orthopedic) as unnecessary and only one student chose each of the other actions (gastrointestinal, respiratory, speech and hearing, and urological). The actions taken more frequently by the TRC counselors than the graduate students were: 1) general medical (94.7% vs 76.9%);2) neurological (84.2 vs 61.5);3) Interview 3 (78.9% vs 69.2%);4) Interview 4 (73.7% vs 23.1%);5) Interview 5 (68.4% vs 7.7%); and 6) Interview 6 (52.6% vs 7.7%).

When the TRC counselors were compared to Berven's original counselor group, none of the counselors in either group chose a cardiovascular examination, gastrointestinal examination, orthopedic examination, respiratory examination, speech and hearing examination, urological examination, or a visual examination. A careful study of the case would indeed indicate that these actions were probably not relevant for determining eligibility. They also chose similar actions, with two notable exceptions. The Texas counselors were more prone to choose the psychological examination over the psychiatric examination (89.5% vs 47.4%) while Berven's group chose the psychiatric examination over the psychological examination (83.3% vs 66.7%). The TRC counselors also used more interviews, with 52.6% using all six while in Berven's study, only 25% opted for all six interviews, and the majority of Berven's counselors did not use more than three interviews. The TRC counselors, with their tendency to request more interviews would appear to be less efficient in their clinical decision making skills than Berven's counselors. It is possible that regional differences in accepted rehabilitation procedures may account for this variance in case management.

Among the criterion group (TRC counselors) in our study, the number of actions taken varied from 4 to 14, with a mean of 9.0 actions (SD = 2.98). Among the undergraduate rehabilitation students, the number of actions taken varied from 2 to 17, with a mean of 9.21 (SD = 3.85). The number of actions taken by the graduate rehabilitation counseling students varied from 3 to 11, with a mean of 6.69 (SD = 2.5). It seemed that the undergraduates had greater variation in the number of actions taken than did the graduate students or the criterion group.

To quantify the performance of individual graduate rehabilitation counseling students relative to the experienced rehabilitation counselors, Berven and Scofield (1980) developed two formulas to evaluate case management performance in terms of two scores, a proficiency index (PI) and an efficiency index (EI).

As an immediate step in computing the proficiency index (PI), it was necessary to compute the proportion of the criterion group that took each action. This was defined as an index of usefulness for that particular action. For example, 91.7% of the experts (rehabilitation counselors) selected Interview 1; therefore, the index of usefulness for Interview 1 was .917. The total usefulness of all available actions was the sum of the usefulness indices over all 21 actions (i.e., .0 + .0 + .917 + .833 + .833 + ... + .333 + .250 = 8.08). The PI for each student was then computed as the sum of the usefulness indices over all actions taken by the student divided by 8.08 and multiplied by 100%. The PI thus yielded a percentage varying from 0% to 100%.

The efficiency index (EI) was computed as the sum of the usefulness indices over all actions taken by the student divided by the total number of actions taken. The EI yielded a percentage ranging from 0% to 91.7% (Berven & Scofield, 1980).

Among the 32 undergraduate students, the PI scores ranged from 19.6 to 100.0 with a mean of 60.6 (SD = 22.3) and the EI scores ranged from 34.8 to 80.6 with a mean of 64.0 (SD = 10.9). Among the graduate students, the PI scores ranged from 27.9 to 86.3 with a mean of 56.4 (SD = 18.3) and the EI scores ranged from 59.4 to 83.4 with a mean of 69.7 (SD = 7.2). The criterion group (TRC counselors) PI scores ranged from 34.1 to 100.0, with a mean of 71.6, which is considered above average (SD = 19.3) and an EI range of 57.8 to 81.3 with a mean of 66.2, which is considered average (SD = 7.1).

A visual examination of the performance data revealed that students varied greatly in their EI scores, PI scores, and number of actions taken. Therefore, a complete-link cluster analysis was performed to separate the students into homogeneous subgroups to characterize their performance more precisely in terms of approaches to problem solving. The proximity (similarity) measure between each pair of students was the number of actions on which the two students agreed, which could range from 0 to 21 for each student pair (Berven & Scofield, 1980). The Cluster procedure of the Statistical Package for the Social Sciences (SPSS/PC +) was used for the computation. (Norusis, 1986)

A three-cluster solution was identified as optimal. Cluster 1 included 18 students who took an average of 5.1 actions with a mean PI score of 42.1 (SD = 16.2) and EI score of 70.6 (SD = 10.8). Cluster 2 included 14 students. They took an average of 7.14 actions with a mean PI score of 59.1 (SD = 9.7) and EI score of 67.9 (SD = 5.8). Cluster 3 was comprised of 14 students who took an average of 11.9 actions with a mean PI score of 81.9 (SD = 12.0) and a mean EI score of 56.9 (SD = 7.7). The students in Cluster 1 reflected what Berven and Scofield (1980) described as the constricted (low proficiency, high efficiency) problem solving approach. Students in this cluster took very few actions to arrive at their decisions about eligibility determination. The students in Cluster 2 reflected what Berven and Scofield (1980) characterized as almost thorough and discriminating problem-solving approach. This cluster also contained the majority of the graduate students (n = 6). The students in Cluster 3 reflected an emphasis on thoroughness or proficiency. This cluster most nearly matched Berven's shotgun approach group.

Berven cautions that the PI and EI indices still need further research validation and should not be definitive measures of case management performance. To facilitate the interpretation of the clusters of the students in terms of actions taken, a step-wise discriminant analysis was performed using the same 21 actions as predictors and group membership devised from the cluster analysis as the criterion variables. The analysis resulted in the extraction of two discriminant functions, the first function accounted for 81.4% of the variance, Wilk's Lambda = .001, [x.sup.2] (28) = 239.4, p [is less than] .001 and the second function accounting for 18.6% of the variance, Wilk's Lambda = .076, [x.sup.2] (13) = 93.0, p [is less than] .001.

The first function was characterized by the use of more interviews and other non-essential actions and thus is labeled as excessive and inappropriate actions. The second function was characterized by more selectivity in actions taken, and use of especially appropriate actions (i.e. neurological examination, work history, etc.) and thus is labeled as more discriminating and appropriate actions. The actions that seemed to differentiate group membership were the selection of only 2 interviews, the neurological examination, and verification of work history.

Cluster 1, the constricted approach, was characterized by low excessive actions, but did not tend to make use of the extra actions which yielded useful information in formulating a decision. Cluster 2, the close to discriminating and thorough approach did have low excess actions but also took those additionally useful actions. Cluster 3, the shotgun approach, did take the extra actions but also was characterized by excessive numbers of actions.

In summary, our study did yield rather similar results to Berven's original study. For example, most of the actions taken by TRC counselors were quite similar to Berven's counselors. In comparing to rehabilitation students (particularly undergraduate students), TRC counselors managed to take more appropriate actions and carefully avoided those absolutely non-essential actions such as a gastrointestinal examination than did the students. The results of this study therefore provided further evidence of criterion validity to Berven's case simulation and support its potential use as an assessment device in rehabilitation education.


Because of the small number of participants in the three comparison groups, the differences between groups observed in this study must be considered with caution and further research will be required to confirm the current findings. Nevertheless, this study further demonstrated the feasibility of using personal computers to provide case simulations in training and evaluating students.

When comparing TRC counselors to Berven's rehabilitation counselors, however, there are two areas of discrepancy. The most striking discrepancy is in the area of number of interviews deemed necessary by the counselors. The majority of Berven's counselors chose to utilize only the first three interviews in the decision making process, while over 50% of the Texas counselors used all six interviews. This may be a reflection of the differing criteria used for counselor selection in the two studies. Berven's counselors all had master's degrees in rehabilitation counseling and were certified by the National Commission on Rehabilitation Counselor Certification (CRCC). Furthermore, they were selected on the basis of having at least 6 years of professional experience, with a mean of 10.5 years (Berven, 1985). In contrast, as a convenient sample, the Texas counselors in our study were selected localy with varied level of training and experience. The majority held a bachelor's degree and the degree was not necessarily in rehabilitation. The level of experience was also not as great as Berven's counselors, with several counselors having been employed as TRC counselors less than one year. These differences in training and experience could well account for the counselors feeling the need for more interview information than needed due to inexperience and lower level of rehabilitation training.

While Berven's counselors chose psychiatric services in a larger number than did the Texas counselors, the Texas counselors chose psychological examinations in larger numbers. This may be a reflection of differing regional practices or differing budget constraints. Since psychiatric services are more costly than psychological services much more extensively than do use psychological services much more extensively than psychiatric, reserving psychiatric examinations for cases that mandate it (i.e., mental health clients).

The practice of using Berven's criterion group as performance standards to compute the PI and EI scores as a feedback mechanism for trainees must be employed with greater caution. The less than perfect agreement in actions taken between the Berven's criterion counselor group and the TRC counselors in this study would suggest that further replication studies may be necessary.

When comparing the choices of the graduate students in our study to both the Texas counselors and Berven's counselors, it can be seen that the graduates were more like Berven's counselors particularly in terms of number of interviews requested. This may again be a reflection of the differing levels of clinical problem solving skills between a master's level and undergraduate level education.

A detailed analysis of the results indicated that under-graduate rehabilitation students participating in study tended to have some difficulty in making proper clinical decisions. The students had difficulty determining the number of actions needed, either requesting too much or too little information. Their choice of actions also showed a lack of discrimination as to which actions were appropriate or inappropriate in serving the simulated case. Therefore, case simulation can be used as a diagnostic tool to determine levels of case management sophistication in terms of clinical problem-solving skills. Students who have not mastered optimal strategies for managing rehabilitation cases can be identified quickly through case simulation, and remedial work can be prescribed.

The use of discriminant analysis to study the action patterns of different decision-making approach groups appear to provide additional meaningful performance information (vs. the use of PI and EI scores alone) and can be easily incorporated into the feedback mechanism of the computer simulation when

it is being used for teaching and not evaluation purposes.

Finally, as pointed out by Chan, Berven, and Lam (in press), the application of computer-based case management simulations in rehabilitation education is still in its infancy. Eventually, simulations may become standard tools for training students in case management skills and in evaluating their skills in functions. Simulations could potentially be used as exercises to practice and refine case management skills and as evaluation procedures in course examinations, comprehensive examinations required for graduate degrees, and credentialing examinations in certification and licensure. Before these potential benefits of computer simulations can be realized, however, continued research and development in several areas will be required. One area is the writing of new case management simulations representing a variety of cases in terms of such features as disability type, case complexity, and information provided. Another area is the development of more sophisticated procedures for the appropriate quantification of performance and to develop a variety of norm groups to use in interpreting performance. Most importantly is perhaps the continuing research on the reliability and validity of case management simulations as assessment devices to document their value in a variety of potential applications.
COPYRIGHT 1989 National Rehabilitation Association
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1989, Gale Group. All rights reserved. Gale Group is a Thomson Corporation Company.

Article Details
Printer friendly Cite/link Email Feedback
Author:Lam, Chow S.
Publication:The Journal of Rehabilitation
Date:Jul 1, 1989
Previous Article:Depression: a primary symptom of Parkinson Disease?
Next Article:Professional fragmentation in rehabilitation counseling.

Related Articles
Computing by committee: sharing searches.
Making a match: inexperienced physician executives and the job market.
Problem solving with general semantics.
Teaching critical thinking online.
Fostering critical thinking skills through a web-based tutorial programme for final year medical students--a randomized controlled study.
Case conceptualization and treatment planning: investigation of problem-solving and clinical judgment.
Case difficulty of simulation software.

Terms of use | Copyright © 2016 Farlex, Inc. | Feedback | For webmasters