What attributes are necessary to be selected for an orthopaedic surgery residency position: perceptions of faculty and residents.

Abstract: Orthopaedic surgery has an extremely competitive residency selection process. The authors discuss which attributes of an orthopaedic surgery residency applicant are the most important in obtaining a position. A comparison of applicants' opinions to those of faculty was also done. Anonymous questionnaires were filled out by orthopaedic surgery residency applicants and faculty orthopaedists at teaching institutions. The most important attributes to obtaining a residency were performance on a local rotation (externship), class rank, and interview performance according to faculty. Applicants thought performance on local rotation, United States Medical Licensing Examination Step 1 scores, and letters of recommendation were the three most important. Both groups ranked research participation, gender, and race as the three least important attributes.

Key Words: applicant attributes, orthopaedic surgery, residency

**********

Obtaining an orthopaedic surgery residency position can be difficult. Orthopaedic surgery has an extremely competitive residency selection process. Several articles have been written that describe the attributes of successful applicants. (1-12) Objective studies have looked at medical school grade point average, Alpha Omega Alpha honor society status, medical school class rank and national board licensing examination scores to see which had positive correlation to obtaining a residency position. (3) Academic performance in orthopaedic residency has been compared with college performance and to performance on standardized tests. (9) The orthopaedic applicant also has been examined from several different points of view. Studies have examined trends concerning minorities and women in orthopaedies. (3,5-8,10-12) Resident applicants also have given their perspective; they liked programs with high morale and good teaching, and tended to dislike programs that were disorganized and treated them poorly on interviews. (2) Research participation often is thought to be advantageous to an applicant, but the amount of research may be misrepresented by applicants. (4)

The goal of the current study was to determine which factors are the most important in obtaining an orthopaedic residency position. Most programs in the United States participate in the match process, and so they do not select residents, they rank them. It seems more appropriate to describe the rank process and not the selection process in the following discussion. The process of ranking a resident applicant usually is not an objective process. Each applicant has several attributes which an individual or program must consider. Objective data about past performance are combined with subjective information such as letters of recommendation, dean's letters, personal statements and interview performance. Faculty who make ranking decisions may use data about an applicant in what manner they choose. To the current authors' knowledge, most residency programs do not have completely objective ways in which to analyze all data and reach decisions about ranking. The current study's goal was to determine which factors faculty consider to be the most important in ranking an applicant. The perceptions of the applicant are compared with those of the faculty.

Materials and Methods

Anonymous questionnaires were sent to the eight teaching institutions in the state of Texas. The military programs were not included. A total of 35 surveys were returned from faculty of the various residency programs. The surveys were sent to the residency directors for distribution to individual faculty, therefore the exact number of potential faculty survey participants is not known. In all, there are approximately 60 faculty members in the eight orthopaedic surgery residency programs in Texas. Residency applicants who interviewed at the University of Texas Medical Branch at Galveston participated in an optional survey with similar questions. The survey was modified only to make it from a residents' perspective. A total of 52 students interviewed, and 46 of them participated in the study (88%). The survey asked participants to rank attributes from most important (score of 1) to least important (score of 10). Table 1 lists the attributes.

Participants then were asked a series of questions about the same core of attributes shown in Table 1. The questions, shown in Table 2, allow each attribute to be considered individually. This allows the reader to examine the attributes one at a time and know what percentage of respondents considered that particular attribute to be important. Questions one to nine were answered with a 1 to 5 ranking (1 = strongly agree to 5 = strongly disagree). Questions 10 to 15 were multiple-choice questions.

The student survey asked some additional questions about how many programs they applied, how many interviews they received, how many interviews they went to and why they choose not to go to some interviews. They also were asked what part of their interview they found to be the most beneficial. Demographic information also was requested. The surveys were collected by a secretary not involved in the selection process, and were not examined until after the match list was created; no postmatch survey was obtained.

Results

The attributes of a residency applicant as shown in Table 1 were ranked on a scale from 1 (most important) to 10 (least important). Table 3 shows the results of the ranking by faculty and by applicants. Faculty ranked performance on externship as the most important factor followed by class rank, and performance on interviews. The remaining categories as ranked by the faculty were United States Medical Licensing Examination (USMLE) Step I as fourth, United States Medical Licensing Examination Step II as fifth, and letters of recommendation as sixth. Applicants had a close grouping for four factors: performance during externship as first, scores on United States Medical Licensing Examination Step I as second, letters of recommendation as third, and medical school class rank as fourth. The remaining categories as ranked by the applicants were performance on interview as fifth, and scores on United States Medical Licensing Examination Step II as sixth. The dean's letter, research participation, gender, and race were ranked 7, 8, 9, and 10, respectively by both faculty and applicants.

Survey participants also were asked a series of questions with a 1 (strongly agree) to 5 (strongly disagree) scoring system. These are questions one to nine from Table 2. These questions allowed an attribute to be considered without regard to other attributes. The results of these questions are summarized in Figure 1. Lower average scores represent agreement with the question while higher scores indicate disagreement.

The first question asked was whether the dean's letter is important in obtaining a residency position. A score of 1 would indicate strong agreement (the letter is important) whereas a score of 5 would indicate strong disagreement (the letter is not important). Scores for the dean's letter averaged 3.1 for faculty and 2.8 for applicants. Whether orthopaedic faculty letters of recommendation are important in obtaining a residency position was the second question. The average score on this question was 2.0 from faculty and 1.7 from applicants. Whether letters of recommendation from non-orthopaedic faculty are important to obtaining a residency position was the third question. The average score on this question was 3.5 from faculty (the highest average noted on these questions) and 2.9 from applicants.

[FIGURE 1 OMITTED]

Performance on externship was considered important to all participants. Question four asked whether it helps their chances of getting a position at an institution when applicants do a rotation at that institution. Question four received an average score of 1.9 by both groups. Whether race plays a role in the ranking process was question five. Most faculty apparently felt that race did not play much of a role, as the average score on question five ("Race does not play a role in the selection process in orthopaedics") was 1.5. The applicants' average score on question five was 2.8, meaning many applicants thought that race does play a role in the ranking process. The role of gender in the ranking process also showed a difference in opinion between faculty and applicants. Question six stated: "Gender does not play a role in the selection process in orthopaedics." Question six was scored at 1.9 by faculty, and at 3.2 by applicants. Therefore most faculty thought that gender does not play a role in the ranking process, but applicants thought otherwise.

Whether research experience is an important criterion for consideration in the application process was treated in question seven. Research participation received an average score of 3.0 by both groups. Whether published research is considered more important than just participation in research was the subject of question eight. The average faculty score on question eight was 2.3 and the average applicant score was 2.4. The importance of an applicant's score on USMLE Step II was question 9. Performance on USMLE Step II received an average score of 2.3 by faculty, whereas applicants thought this was less important, with an average score of 3.3.

The remaining questions were multiple choice, and the results are included in Figures 2 through 4. Figure 2 summarizes opinions concerning class rank as an attribute. Faculty opinions on what an applicant's medical school class rank should be to obtain a position are as follows: 3% choose top 5% of class, 6% choose top 10% of the class, 34% choose top 20% of the class, 44% choose top 30% of the class and 13% choose the top 40% of the class. More than 85% of faculty chose top 30% of medical school class or higher as a minimal standard required to obtain a residency position. Applicant opinions of class rank were slightly different. No applicant thought top 5% of the class was necessary and 12% choose top 10% of the class. The top 20% of the class was selected by 36% of applicants, the top 30% of class was chosen by 27% of applicants and top 40% of the class was chosen by 25% of applicants. Therefore the majority (75%) of applicants also thought a ranking in the top 30% of medical school was required to obtain an orthopaedic residency position.

When medical students are deciding about residency positions they often are concerned they may not even get interviews for orthopaedic positions. Applicants and faculty were in relative agreement about minimal class rank needed to obtain an interview. No participant thought a class rank of top 10% or higher was required to obtain an interview. A top 20% of the class ranking was considered necessary by 16% of faculty and 22% of applicants. The top 30% of the class was selected by 22% of faculty and 20% of applicants. The majority of participants thought the top 40% of the class should be able to obtain interviews, with 62% of faculty and 58% of applicants choosing this level of performance.

[FIGURE 2 OMITTED]

Scores on USMLE Step I are available for the ranking process. Figure 3 summarizes opinions about minimal scores an applicant should receive to be successful at obtaining a residency position. Scores below 200 were thought to be too low by both groups. Scores of 201 to 220 were selected by 29% of faculty and 14% of applicants. A score in the 221 to 230 range were selected by 48% of the faculty and 36% of the applicants. A 231 to 240 score was thought to be necessary by 16% of faculty and 41% of applicants. Scores above 240 were thought to be necessary by 7% of faculty and 9% of applicants. Half of the applicants thought a USMLE Step I score of 231 or higher was required, whereas only 23% of faculty thought scores this high were required.

Both groups indicated that applicants with lower USMLE Step I scores still could receive interviews. Thirty-five percent of faculty and 28% of applicants thought an applicant with a score below 200 still should get an interview. The range from 201 to 220 was that most commonly thought by both groups to be the minimal needed to obtain an interview (faculty 46%, applicants 42%). A score of 221 to 230 was selected by 19% of faculty and by 28% of applicants. Only 2% of applicants thought scores above 230 were required. No faculty thought a score above 230 was required to obtain an interview.

[FIGURE 3 OMITTED]

[FIGURE 4 OMITTED]

Figure 4 shows the results of participant opinions about what score on USMLE Step II an applicant should receive if they are to obtain a residency position. Step II scores usually are not available before interviews are granted, so a minimal score to obtain an interview was not discussed. Scores below 200 were considered acceptable by 4% of applicants and 0% of faculty. Scores of 201 to 220 were selected by 33% of faculty and by 17% of applicants. Scores of 221 to 230 was the most common range cited by both groups, with 50% of faculty and 44% of applicants choosing this range. Scores of 231 to 240 were selected by 12% of faculty and by 29% of applicants. Scores above 240 were thought to be necessary by 5% of faculty and 6% of applicants.

Applicants in general thought it was necessary to apply to more programs than did faculty. Four percent of applicants though they needed to apply to more than 80 programs, and another 8% thought they needed to apply to 60 to 80 programs. No faculty thought applying to more than 60 programs was necessary. Only 7% of faculty thought 41 to 60 applications was appropriate, whereas 47% of applicants chose this range. Fifty-six percent of faculty recommended application to between 21 and 40 programs, while 26% of applicants thought this was the best number of applications. Thirty-seven percent of faculty recommended application to 20 programs or less, whereas 15% of applicants chose this number of programs.

The remaining results regarding the application and interview process were obtained from the applicants only. The most common reason for application to a program was location. Faculty recommendation or recommendations from residents or other applicants were used by 38% of applicants to make application decisions. Approximately 60% of the applicants received 11 to 20 interviews, and most of them wanted to go to all of these interviews. Conflicting dates and travel costs were listed as the two most common reasons for not being able to attend an interview. The most beneficial part of the interview process was talking to the residents at a program, according to 92% of the applicants. Seven percent thought that talking with faculty was the most important part of their interview, and only 1% thought the hospital tour was most important.

Discussion

For applicants to obtain competitive residency positions it is important for them to know what their chances are. Medical students may benefit from knowing what characteristics are considered to be the most important. A bias could have occurred in this study because only 35 of the 60 possible faculty responded to the survey. Responding took time and effort, and the trends discussed may represent the views of only the more involved faculty members. It should also be noted that the survey took place in a geographically limited area, and may not be representative of the entire national residency selection process. The authors thought it also would be useful to residency programs to know what applicants are thinking.

The principle of a "minimum standard" is used in this article with regard to USMLE scores and class rank. Minimal scores and ranks that are required to be granted an interview and those required to match are discussed. For purposes of this study each applicant attribute, including the score and rank, was examined individually--that of course is not how applicants are granted interviews or ranked. Most applicants are evaluated on the basis of all of their attributes. Therefore, each attribute probably has variable importance dependent upon other factors. For instance, if an applicant's USMLE step 1 score is low or at a minimal level, then other factors may become relatively more important in evaluating that applicant. So an applicant with a low USMLE score may still be granted an interview or ranked higher if other factors are outstanding.

The research presented makes it apparent that applicants and faculty agree on many factors. There are some areas where faculty and applicants do not agree. Applicants give nearly equal weight to performance on externship, USMLE Step I scores, letters of recommendation, and class rank. Externship performance clearly was the most important characteristic faculty chose, followed by class rank, and performance on the interview. This may suggest faculty wanted to know the applicant better because many applicants have similar attributes on paper.

The current results suggest the dean's letter is not one of the more important criteria used in the ranking of residents. Letters of recommendation are important, with letters from orthopaedic faculty being more important than those of other specialties. Faculty experience may have some affect on the importance given to letters of recommendation. About 60% of the total faculty responded to this survey, and the demographics/work experience of these respondents is unknown. More experienced faculty might be expected to be more familiar with the people writing these letters, and to have seen more letters. The more experienced faculty could therefore be expected to be able stratify or rank these letters, and therefore give them more relative importance. There was slight disagreement between faculty and applicants on the importance of letters of recommendation.

Research participation was not regarded as highly as might have been anticipated. The reader is cautioned about this point because it is clear that the Accreditation Council for Graduate Medical Education and the Residency Review Committee for orthopaedic surgery (1) are recommending and encouraging more resident research. If this study were to be repeated in 3 to 5 years it is likely that research participation would be regarded as a highly important variable. Research participation likely also has variable importance depending upon other factors. Each residency program is, however, different. Larger programs with better research resources often mandate research and may place more of an emphasis on this during interviews and selection. Some programs emphasize research and others do not; the statement that research participation will become more important in the future may reflect the biases of the authors. The authors' residency program recently has mandated research requirements with the expectation that all residents will publish articles during residency.

Race and gender were considered to be of low importance when compared with other attributes. When examined without regarding other attributes, many applicants thought that race and gender were considered in the selection process. Applicants who were female or non-white, thought more strongly that these two factors were considered in the ranking process. So race and gender may have more effect on the subjective thoughts of the applicant.

No attempt was made in the current study to do statistical analysis between variables. Some of the variables studied were objective, such as scores, while others, eg, letters of recommendation, were subjective. In some programs even these subjective factors, like letters of recommendation, can be made more objective if they are rated (ie, excellent, good, fair, poor or 1-5). The selection process includes subjective and objective criteria, but the ultimate ranking of a resident applicant usually is subjective. The authors were attempting to provide applicants with information they may find useful in pursuit of a residency. The authors acknowledge that faculty perceptions may not always be reality. Faculty may have perceptions about which resident attributes are important, but there is no way to know if these perceptions are employed when ranking a candidate. Previous studies have performed statistical analysis on resident attributes to determine which ones correlated to success in obtaining a residency position. (3) That was not the objective of this study. The goal of the current study was to have faculty and applicants rank the attributes in terms of importance. Therefore, a limitation of the current study is that perceptions and opinions are not validated. Despite this, the authors believe this study offers a reasonable method to compare residency applicant attributes. This will provide applicants with some guidance on where to concentrate their efforts. Applicants should perform well on externships and concentrate on the interview to help with the subjective portion of the ranking process. Performance in medical school and on standardized examinations is very important. Those applicants with low scores in school and on tests may find it difficult to even obtain an interview, and even more difficult to obtain a residency. This information should help in career selection and guidance of applicants.

Program directors should take note that applicants think the most important part of the interview day is talking with residents. Residents and applicants should be encouraged to interact on interview day. It is not the intent of this research to recommend to program directors what they should consider important or to standardize the ranking process. Each program is unique and therefore the ranking process never can be standardized. Also not discussed is the importance of informal communications. Phone calls, e-mails, and follow-up letters after interviews all have some effect on the ranking process. Program coordinators and other individuals that set up interviews are likely to have numerous contacts with applicants. These informal contacts can have some effect on the interview and ranking process so applicants are encouraged to remember that all contacts with a program are important.

The variations between programs produce further bias about which attributes are important. The authors do not hope to imply that this data can be used for every program. This was a geographically limited study. Large programs with multiple financial resources may have different values than small financially constrained programs. Research requires financial support and may be considered more important in larger programs. Historic success in the match also affect what attributes are needed for interviews and rankings. Programs that traditionally fill all of their slots with only top ranked applicants may focus on certain criteria unique to their program knowing they are likely to get their top choices.

The authors' residency program provides an example of objective data about the selection process. Approximately 450 applications are received for 4 residency positions. Interviews are granted to approximately 50 applicants. Interviews are granted to applicants with the highest USMLE scores, those who are in the Alpha Omega Alpha honor society or are in the top of their medical school class, and those who have some research experience. Other applicants are granted interviews at the discretion of the program director or the application committee. Approximately 50 applicants are ranked each year at the authors' institution. The National Residency Match Program provides data on where each ranked applicant found a residency position or if they did not find a position. Applicants from the previous two years were examined. The average USMLE Step I for applicants who were ranked at the authors' institution and then matched in any United States orthopaedic surgery residency was 236 (n = 91). The average score for those applicants who did not match was 215 (n = 8). Seventy-two percent of applicants who matched were in the Alpha Omega Alpha honor society (applicants who were from schools that did not offer this society are not included in this percentage calculation) while 12.5% of applicants who were unsuccessful were in the Alpha Omega Alpha honor society. These objective trends are from one institution and therefore cannot be generalized. A large-scale objective examination of residency attributes is left to future studies.

Conclusion

The current research is part of an ongoing evaluation of the residency application and selection process. Future research will examine the objective differences between successful and unsuccessful applicants. Another area of research will include examination of how these preresidency variables correlate with performance during residency and on national board examinations. The specialty of orthopaedics has become increasingly popular during the past decade. The number of applicants exceeds the number of available residency positions, thereby producing intense competition. Therefore, orthopaedic programs are in a position to be very selective in their appointments and rankings for residency. It is important for applicants and faculty to understand how applicants are ranked in order to continue to improve the process.

Table 1. Residency applicant attributes

Class rank
USMLE Step 1 score
USMLE Step 2 score
Dean's letter
Letters of recommendation
Performance on externship rotation
Race
Gender
Research experience
Interview evaluation

Table 2. Areas of importance in judging an orthopaedic applicant

 1. The dean's letter is important in obtaining an orthopaedic residency
 2. Orthopaedic faculty letters of recommendation are important in the
    application process
 3. Non-orthopaedic faculty letters of recommendation are important in
    the application process
 4. Applicants who do a rotation at your institution helps their chances
    of getting a position at your institution
 5. Race does not play a role in the selection process
 6. Gender does not play a role in the selection process
 7. Research experience is an important criterion for consideration in
    the application process
 8. Published research is considered more important than just
    participation in research
 9. The score of an applicant on USMLE Step II is important
10. Minimum class rank to accept an applicant (top 5%, 10%, 20%, 30%,
    40%)
11. Below what rank should interview not be granted (top 5%, 10%, 20%,
    30%, 40%)
12. What USMLE Step I scores are needed to obtain a residency position
    (180-200, 201-220, 221-230, 231-240 and 240+)
13. Below what USMLE Step I score should an applicant not be granted an
    interview (same range as question 12)
14. What USMLE Step II score is needed to obtain a residency position
    (same range as question 12)
15. How many programs do you believe an applicant should apply (1-20,
    21-40, 41-60, 61-80, 81+, or does not matter)

Table 3. Results of faculty and applicant ranking of resident attributes

Attribute            Faculty rank  Raw score  Applicant rank  Raw score

Performance
  on externship       1             93         1              158
Class rank            2            107         4              164
Interview
  performance         3            112         5              174
USMLE Step I score    4            123         2              159
USMLE Step II score   5            151         6              218
Letters of
  recommendation      6            153         3              160
Dean's letter         7            166         7              251
Research
  participation       8            203         8              283
Gender                9            230         9              414
Race                 10            261        10              428

Raw score is compiled by multiplying number of participants who ranked
that attribute by the rank they gave the attribute and then adding up
those numbers. The lower the raw score the more important that
attribute. Eg. if 10 participants listed class rank as #1 and the 25
others ranked it #2, then score would be (10 X 1) + (25 X 2) = 60 raw
score. Higher raw scores for applicants are because of higher numbers of
participants.

Accepted June 1, 2004.

Please see Angus McBryde's editorial on page 1154 of this issue.

References

1. Accreditation Council for Graduate Medical Education. Graduate Medical Education Directory 2001-2002, AMA 2001.

2. Buch WH, Champman RG, Dvonch VM. The candidate's view of the orthopaedic residency selection process. J Bone Joint Surg 68A:1292-1296.

3. Clark R, Evans EB, Ivey FM, et al. Characteristics of successful and unsuccessful applicants to orthopedic residency training programs. Clin Orthop 1989;241:257-264.

4. Dale JA, Schmitt CM, Crosby LA. Misrepresentation of research criteria by orthopaedic residency applicants. J Bone Joint Surg 1999;81A:1679-1681.

5. England SP, Pierce RO. Current diversity in orthopaedics. Clin Orthop 1999;362:40-43.

6. Grant RE, Banks Jr WJ, Alleyne KR. A survey of the ethnic and racial eistribution in orthopedic residency programs in the United States. J Nat Medical Assoc 1999;91:509-512.

7. Jimenez RL. Barriers to minorities in the orthopaedic profession. Clin Orthop 1999;362:44-50.

8. Phillips PJ. Barriers to minority participation in the orthopaedic profession. Clin Orthop 1999;362:51-54.

9. Ronai AK, Golmon ME, Shanks CA, et al. Relationship between past academic performance and results of specialty in-training examinations. J Medical Educ 1984;59:341-344.

10. Scherl SA, Lively N, Simon MA. Initial review of electronic residency application service charts by orthopaedic residency faculty members does applicant gender matter? J Bone Joint Surg 2001;83A:65-70.

11. Thomas CL. African Americans and women in orthopaedic residency. Clin Orthop 1999;362:65-71.

12. Walker JL. Janssen H, Hubbard D. Gender differences in attrition from orthopaedic surgery residency. J American Women Medical Association 48:182-184.

RELATED ARTICLE: Key Points

* Competition is stiff among candidates for residency positions in orthopaedic surgery.

* Faculty believe performance during externship, class rank, and interview performance are the three most important factors in applicant selection.

* Applicants believe performance during externship, United States Medical Licensing Examination Step I score, and letters of recommendation were the three most important factors.

* Both applicants and faculty rated research, gender, and race as being of least importance.

Gurpreet Bajaj, MD and Kelly D. Carmichael, MD

From the University of Texas Medical Branch School of Medicine, Department of Orthopaedics and Rehabilitation, Galveston, TX.

Reprint requests to Kelly D. Carmichael, MD, 301 University Blvd. Route 0353, Galveston, TX 77555. Email: [e-mail]