Relation of residency selection factors to subsequent Orthopaedic in-training examination performance.

Objectives: Orthopaedic surgery remains one of the most competitive specialties, with more than a 99% match fill rate in the past several years. An oversupply of qualified applicants leads to intense competition for these residency spots, allowing program directors to be more selective in choosing their future residents. Although many previous studies have documented factors important to program directors in the admission process, less is known about how preselection factors correlate with subsequent performance in a residency program.

Methods: The relation of both demographic and academic factors with subsequent performance on the Orthopaedic In-Training Examination (OITE) were studied. These factors include United States Medical Licensing Examination (USMLE) step I scores, Alpha Omega Alpha (AOA) status, research publications, age entering residency, marital status, and medical school affiliation.

Results: In this study, the only statistically significant correlations to OITE scores were USMLE step I performance and marital status. Those residents who had previously scored above 220 on the USMLE step I had higher average OITE scores than those scoring below 220. Residents who were married also had higher average OITE scores. A trend with regard to AOA status also was found, with residents scoring slightly higher on the OITE if they were members of AOA.

Conclusions: Few preresidency variables correlate to success during an orthopaedic residency.

Key Words: Alpha Omega Alpha, Orthopaedic In-Training Examination, orthopaedic surgery, resident success, United States Medical Licensing Examination

**********

The specialty of orthopaedic surgery remains an extremely competitive field in which to match. Although only two thirds of positions filled in the National Resident Matching Program matched 25 years ago, more than 99% of residency positions have been filled in recent years. (1) This trend obviously reflects the increasing popularity of orthopaedic surgery as a postgraduate specialty. It also is a testament to the competition for these positions. The number of available positions is vastly less than the number of applicants. In fact, in 2000, there were 1, 116 formal applicants for the 554 PGY-1 spots in the specialty. (2)

The oversupply of qualified applicants leads to intense competition for these spots. In practice, this means that residency directors can afford to be selective in choosing future residents. Prior studies have suggested that as a specialty becomes more competitive, the associated residency programs will rely more on academic credentials for screening. (1,3) The characteristics of successful orthopaedic surgery applicants have been examined, and markers of academic prowess have correlated well with successful applicants. (1,2,4) A few studies have tried to correlate preresidency variables to subsequent performance in residency. Although college grade point average (GPA) and Medical College Admission Test (MCAT) scores may be markers for preclinical success, they do not appear to correlate well with success during clinical activities. (5) In-training examination success has been shown to have some correlation to National Board of Medical Examiners (NBME) scores. (10)

It seems clear that the academic success of a resident applicant to Orthopaedic Surgery factors into the selection process. Do any preresidency factors correlate to success during residency? And what is success during a residency? A previous study has attempted to correlate preresidency variables to success during residency. No predictor was found for Orthopaedic In-Training Examination (OITE) or American Board of Orthopaedic Surgery part 1 examination success. Faculty ranking of resident performance was, however, linked to medical school clerkship performance and to a lesser degree Alpha Omega Alpha (AOA) status. (6) Performance in residency depends on several subjective criteria as well as objective criteria. For study purposes, we have chosen the OITE as the defined outcome measure for the different predictors. In practice, the OITE is used as an instrument to measure resident progress. The OITE is not designed to be used internally or externally as an outcome measure. We have examined several preresidency factors, some academic and others demographic, to see if there is a correlation between these variables and success on the OITE. Performance on the OITE is also compared with faculty perceptions of resident performance.

Materials and Methods

We examined average scores for the OITE among 60 residents in two Texas programs, from 1994 to 2002. All residents are included in this study to their present level, from PGY-1 to PGY-5. The senior author obtained the scores. Each resident had one to five scores, depending on the number of years that they had spent in training. All scores were combined into a master list, and an average OITE score was calculated for each resident. The average scores were then compared with preresidency variables. These variables included United States Medical Licensing Examination (USMLE) step I score, AOA status, marital status, sex, medical school affiliation, age at beginning of residency, and prior research experience. The list of variables studied is not all-inclusive. There are many other factors that are likely to influence performance that were not studied. The OITE scores were placed in the appropriate categories as indicated, and each possible relation was treated as if it were an independent variable. Mean value and range was found for each column, and statistical analysis was performed to see if any significant differences existed between the subgroups, the results of which are presented below.

We also tried to correlate faculty perception of performance to preresidency variables. We asked faculty at our institution to rank each class of graduating residents. No set criterion was given in which the residents should be rated, so faculty could use any method they chose to rank each class of residents from 1 (best) to 4 (worst). Only faculty who were employed for the entire residency of the five classes were used. This provided 9 faculty members and 20 residents of 5 graduating classes. Each of the five No. 1 rated residents had their preresidency variables examined, as did the five No. 2, No. 3, and No. 4 rated residents. The number of females was too low to permit gender discussions for privacy reasons. The survey was done over five consecutive graduating classes, the years of which are excluded also for privacy reasons.

Statistical analyses were performed on the data using PC-SAS (SAS Institute, Cary, NC) An analysis of variance using the Proc general linear model was used to test for differences in OITE scores due to the independent variables. Unless explicitly stated otherwise, in this report, a probability value of less than 0.05 was considered to be statistically significant.

Results

The correlation between USMLE step 1 scores and average OITE percentile is summarized in Table 1. The majority of residents (52%) had a USMLE score between 221 and 240. Scores of 201 to 240 accounted for 83% of the residents, with scores below 200 (12%) and those above 240 (5%) being the minority. Residents with scores of 200 or below (n = 7 or 12%) had an average OITE score of 54. Residents with USMLE scores between 201 to 220 (n = 19 or 32%) had an OITE average of 51. Residents with USMLE step 1 scores of 221 to 240 (n = 31 or 52%) and those above 240 (n = 3 or 5%) had OITE score averages of 63 and 87, respectively. Although a trend for higher OITE scores with higher USMLE scores existed, there were not any statistically significant differences among the OITE scores noted between the groups.

There were relatively small numbers in each USMLE score range when using four groups. The groups were combined to make two ranges, and the results are summarized in Table 2. Residents with USMLE step 1 scores at or below 220 (n = 26 or 43%) had an average OITE score of 51, whereas those above 220 (n = 34 or 57%) had an average of 66. There was a statistically significant difference (P = 0.04) between these two groups indicating residents who score above 220 on the USMLE step 1 are statistically more likely to have higher OITE scores than residents with scores at or below 220.

Membership in AOA society was then correlated to OITE scores, with the results summarized in Table 3. Residents who were members of AOA (n = 29 or 48%) had an average OITE score of 64, whereas those who were not members (n = 31 or 52%) had an average score of 55. There was not a statistically significant difference between these groups. There was a trend toward significance (P = 0.09) between these groups, with members of AOA scoring higher than those who were not members.

Having prior research experience with publications was examined to see if this factor correlated to OITE scores. Research publication data were not available from one of the institutions, so the total number of participants in which this factor was known is 32. Residents who had prior publications (n = 8 or 25%) and those without prior publications (n = 24 or 75%) had average OITE scores of 55 and 61, respectively. There was not a statistically significant difference between these groups (Table 4).

Marital status entering residency was examined to see if this had any effect on performance. The results are summarized in Table 5. Residents who were married (n = 40 or 67%) and those who were not (n = 20 or 33%) had average OITE scores of 64 and 51, respectively. There was a statistically significant difference (P = 0.012) noted between these groups, with married residents scoring higher on the OITE.

Medical school attendance was examined. Having direct knowledge of residency applicants is thought by some to confer an advantage on that applicant. Do applicants familiar with a residency (those from the same institution) have an advantage? Residents were divided into those who attended medical school at the same institution as residency and those that attended another medical school outside their residency institution. The results are summarized in Table 6. When medical school and residency institution were the same (n = 20 or 33%), the average OITE score was 59. When medical school and residency were done at separate institutions (n = 40 or 67%), the average OITE score was 60. There was not a statistically significant difference between these groups.

Age entering residency was also examined, with the results summarized in Table 7. Residents who were older than 27 years (range, 28 to 39 years; n = 24 or 42%) at the beginning of residency were compared with those who were 27 years old or younger (range, 23 to 27 years old; n = 33 or 58%). The age of three of the residents was not known, leaving 57 residents for this comparison. The older residents had an average OITE score of 61 compared with 59 for the younger residents. There was no statistically significant difference between these two groups.

There were 56 male residents (93%) and 4 female (7%) residents. The number of females is too low to permit discussion of OITE averages. There was not a statistically significant difference between males and females.

Faculty members (n = 9) at our institution were asked to rank each of the residents from five consecutive graduating classes from 1 (best resident of that class) to 4 (worst resident of that class). The faculty could rank these residents using any criteria they choose. The rankings were then correlated to USMLE, AOA, age, marital status, publications, whether they were from our institution. OITE average, and American Board of Orthopaedic Surgery (ABOS) part 1 passing rate. The results are summarized in Table 8.

The numbers of residents involved in this faculty survey are low and do not permit statistical analysis. In this very limited example, faculty perception of the best resident of a class did not correlate with any of the other factors. The No. 1 ranked residents and the No. 4 ranked residents all had similar nonacademic traits and had similar USMLE and OITE scores. ABOS part 1 passing rate was high in all groups, but an unexpected finding was that the only residents to fail part 1 of the ABOS examination were selected by faculty as the best resident in their class.

Discussion

There have been many papers written over the years on the residency application process, both in general and specific to the specialty of orthopaedic surgery. These studies have examined trends in admissions as well as the factors, which were most important in obtaining admission to a particular specialty. Clark et al (1) found that higher NBME scores, higher class rank, and membership in AOA correlated well with admission into an orthopaedic residency program. Surveys of program directors have revealed similar trends of reliance on past academic performance in the evaluation of residency applicants. (2,7) Wagoner and Suriano (3) have noted that as a specialty becomes more competitive, the associated residency directors will tend to rely more on academic credentials when screening applicants. It is obvious that academic prowess is important to the specialty to amass an adequate knowledge base. In addition, it is to the advantage of programs to admit individuals who are capable and likely to pass the ABOS step I. Each program is encouraged to maintain a 75% pass rate on the examination to maintain accreditation, according to the residency review committee.

Perhaps it is more difficult to evaluate the outcomes of using such selection criteria in choosing orthopaedic residents. Some have argued that the admissions process should in effect agree on the type of orthopaedic surgeon that the specialty would like to produce and work to identify characteristics that this individual would possess. (8) Reede (5) found that whereas college GPA and MCAT scores had some correlation to preclinical success in medical school, these markers could not predict clinical success in medical school, residency, or in practice. Therefore, in the context of minority applicants, reliance on premedical academic success may not predict future success. Dirschl et al (6) attempted to correlate residency selection characteristics with subsequent faculty evaluations as well as performance on the OITE and ABOS part I. The faculty members were asked to evaluate the residents in psychomotor, affective, cognitive, and overall performance. Clinical clerkship performance as indicated by honors grades was found to have the highest correlation to overall resident performance, and AOA status was second. Scholastic Aptitude Test scores were the only correlate to OITE scores, and no marker correlated with ABOS part 1 success. (6) Faculty ratings were fairly consistent between the faculty members. Faculty rating in this study as in ours, however, showed poor correlation to OITE and ABOS part 1 scores. (6)

Defining resident success is a difficult task. Faculty rankings, even if the criteria for evaluation is defined, are subjective. The OITE is a more objective way to define academic success. The OITE is administered to all residents from PGY-2 to PGY-5 in a consistent manner. Scores are given as a raw score and as a percentile ranking for each year in training. A limitation of this study is that the OITE percentile is used as the measure of success. In practical terms, a difference of 10 percentile on the OITE may represent a difference of only 3 or 4 questions out of the 275 questions that comprise the test. In analyzing our results, we have attempted to evaluate seven independent variables for their relation to OITE scores. This is the main goal of our study. Subjective faculty rankings are also included in this study. It may be the task of others to correlate success on the in-training examination to success as an orthopaedic surgeon. The data studied include ranges of USMLE scores, AOA status, prior research publications, marital status, medical school attended (same versus other), sex, and age entering residency. Statistical analysis revealed significance among two groups, USMLE scores and marital status.

Among USMLE scores, there was a significant difference in scores based on whether or not the resident scored above or below 220. Marital status was found to be associated with OITE success. Those residents who were married at the beginning of their residency had a higher average on the OITE than those who were single. The sample size was larger for the married group, 40 married versus 20 single residents. The relevance of this is uncertain but is perhaps related to a more regimented life and perhaps blocked study time. However, one could argue that individuals who are married may have much more in the way of time commitments. We did not examine length of time married or the presence of children, both of which might have some influence on performance. A trend was also present with regard to AOA status. Members of AOA had slightly higher average OITE scores than those who were not members.

The prior research experience of an applicant was not an indicator of future OITE success. In fact, the average OITE score was lower for those with documented publication, but this was not statistically significant. Reasons for the negative correlation of research to OITE success are not entirely clear. Perhaps in some cases research is done by applicants who have less competitive credentials in an effort to bolster their chances. In other cases, residents who did not match one year may spend an extra year doing research and then reapply. Studies by Dale et al (9) have shown that many applicants are not completely honest about the extent of their involvement in research. We therefore used prior publications to indicate preresidency research involvement. Finally, residents who went to medical school at the same institution as their residency seem to have similar OITE success as those from other medical schools.

The level of competition for orthopaedics necessitates a careful and stringent screening process. Success as a resident is difficult to define and difficult to predict. Our study shows the highest correlation for OITE success to be performance on the USMLE step 1 score and marital status. Residents with scores above 220 on USMLE step 1 and married residents had higher average OITE scores in this study. We realize, however, that our study does not involve a very large cohort and that it represents a small geographic area in only one state. It must be further noted that we do not believe the OITE score is the only measure of a successful resident. The results of our faculty survey also show that OITE scores do not correlate well to faculty perception of resident performance; the consensus was that the best residents of a given class had OITE scores similar to the lowest rated residents. A reliable model for predicting success during residency has yet to be developed. In fact, the definition of a successful resident is not clear. The reliance on ranking applicants based exclusively on academic success and test scores may not be a good way to obtain the best residents. The residents who score the highest on the OITE may not be the best residents and may not have the most success during their practices.

What loneliness is more lonely than distrust?
--George Eliot, Middlemarch

Table 1. USMLE step I scores group comparison to OITE average (a)

USMLE                               Mean OITE
score range                         percentile

[less than or equal to]200 (n = 7)  54 [+ or -] 22.6
201-220 (n = 19)                    51 [+ or -] 14.9
221-240 (n = 31)                    63 [+ or -] 17.9
> 240 (n = 3)                       87 [+ or -] 13.5

(a) USMLE, United States Medical Licensing Examination; OITE,
Orthopaedic In-Training Examination.

Table 2. USMLE step I scores combined into two groups comparison to OITE
average (a)

USMLE score    Mean OITE
range          percentile

<221 (n = 26)  51 [+ or -] 16.5
>220 (n = 34)  66 [+ or -] 18.7

(a) USMLE, United States Medical Licensing Examination; OITE,
Orthopaedic In-Training Examination.

Table 3. AOA status comparison to OITE average (a)

                     Mean OITE
AOA status           percentile

AOA member (n = 29)  64 [+ or -] 18.8
Nonmember (n = 31)   55 [+ or -] 18.6

(a) AOA, alpha omega alpha; OITE, Orthopaedic In-Training Examination.

Table 4. Research/publications comparison to OITE average (a)

                                        Mean OITE
Status of prior research                percentile

Prior research publication (n = 8)      55 [+ or -] 15.3
No prior research publication (n = 24)  61 [+ or -] 19.9

(a) OITE, Orthopaedic In-Training Examination.

Table 5. Marital status comparison to OITE average (a)

Marital status entering      Mean OITE
residency                    percentile

Married (n = 40)             64 [+ or -] 19.1
Single or divorced (n = 20)  51 [+ or -] 16.0

(a) OITE, Orthopaedic In-Training Examination.

Table 6. Medical school attended correlation to OITE average (a)

Medical school                 Mean OITE
attended                       percentile

Same as residency (n = 20)     59 [+ or -] 20.4
Other medical school (n = 40)  60 [+ or -] 18.4

(a) OITE, Orthopaedic In-Training Examination.

Table 7. Age correlated to OITE average (a)

                              Mean OITE
Age                           percentile

27 or less (n = 24 or 42%)    59 [+ or -] 20.8
More than 27 (n = 33 or 58%)  61 [+ or -] 18.4

(a) OITE, Orthopaedic In-Training Examination.

Table 8. Faculty ranking of resident correlation to preresidency
variables (a)

             AOA,  Average  Married,  Publications,  From        ABOS
Rank  USMLE   %     Age       %           %          UTMB  OITE  pass

1     222    80    28        60           40         40     61    60
2     211    40    31        80            0         80     70   100
3     226    60    29       100           40         40     67   100
4     221    40    30        60           40         60     61   100

(a) USMLE, United States Medical Licensing Examination; AOA, Alpha Omega
Alpha; UTMB, University of Texas Medical Branch; OITE, Orthopaedic
In-Training Examination; ABOS, American Board of Orthopaedic Surgery
part 1 exam.

Accepted December 16, 2004.

References

1. Clark R, Evans EB, Ivey FM, et al. Characteristics of successful and unsuccessful applicants to orthopaedic residency training programs. Clin Orthop Rel Res 1989;241:257-263.

2. Bernstein AD, Jazrawi LM, Elbeshbeshy B, et al. Orthopaedic resident selection criteria. J Bone Joint Surg 2002;84:2090-2096.

3. Wagoner NE, Suriano JR. Factors used by program directors to select residents. J Med Educ 1986;61:10-21.

4. Wagoner NE, Suriano JR. Recommendations for changing the resident selection process based on a survey of program directors. Acad Med 1992;67:459-465.

5. Reede JY. Predictors of success in medicine. Clin Orthop Rel Res 1999;362:72-77.

6. Dirschl DR, Dahners LE, Adams GL, et al. Correlating selection criteria with subsequent performance as residents. Clin Orthop Rel Res 2002;399:265-271.

7. Simon MA. The education of future orthopaedists: deja vu. J Bone Joint Surg Am 2001;83A:1416-1423.

8. White AA. Resident selection: are we putting the cart before the horse? Clin Orthop Rel Res 2002;99:255-259.

9. Dale JA, Schmitt CM, Crosby LA. Misrepresentation of research criteria by orthopaedic residency applicants. J Bone Joint Surg 1999;81A:1679-1681.

10. Ronai AK, Golmon ME, Shanks CA, et al. Relationship between past academic performance and results on specialty in-training examinations. J Med Educ 1984;59:341-344.

RELATED ARTICLE: Key Points

* Few preresidency variables correlate to success during an orthopaedic residency.

* Age, sex, Alpha Omega Alpha status, research, and medical school affiliation do not correlate with subsequent success on orthopaedic in-training examination scores.

* Having United States Medical Licensing Examination scores above 220 and being married did correlate to higher Orthopaedic In-Training Examination percentile scores.

Kelly D. Carmichael, MD, James B. Westmoreland, MD, John A. Thomas, MD, and Rita M. Patterson, PHD

From the University of Texas Medical Branch, Galveston, TX.

Reprint requests to Dr. Kelly D. Carmichael, 301 University Boulevard, Galveston, TX 77555-0353. Email: [e-mail]