Candidate's questionnaire: an alternative to an admissions interview for applicants to a dental hygiene program.

ABSTRACT

Admissions committees are challenged to select the best applicants from a pool of qualified candidates. Requirements for admission to a Canadian dental hygiene program have included required university subjects average (RSA), average of the last five credits taken at a university level, a structured interview, and a candidate's questionnaire. In 1996, the School of Dental Hygiene at Dalhousie University made a decision to use a candidate's questionnaire (CQ), designed to assess the applicant's knowledge of the program and the profession, instead of an interview because the latter was labour intensive and logistically difficult. To investigate how applicants with low academic prerequisites and high CQ scores performed in the program, the questionnaire scores were collected and compared with performance as indicated by overall grade point average (GPA) at the end of the first year in the program. In their first year, among students with high RSA, those with medium or high CQ perform significantly better than those with low CQ (p<0.01). Among students with low RSA ([less than or equal to]72.5%), those with high or low CQ perform significantly better than those with medium CQ (p<0.01, p<0.05).

These results suggest that prior knowledge of the profession and the program influences students' performance levels in the first year of the program. Students who have a better understanding of the profession and the program for which they are applying overcome any disadvantage presented by low academic ability, as suggested by a low RSA. This study suggests that a candidate's questionnaire may be a reasonable tool to use when deciding which applicants with low academic requirements to admit to the program.

RESUME

Les comites d'admission ont une difficulte a surmonter : celle de choisir les meilleurs parmi un groupe de candidats qualifies. Auparavant, pour admettre une personne dans un programme d'hygiene dentaire au Canada, on se basait sur la moyenne requise dans les matieres universitaires (MRMU), la moyenne des cinq derniers credits de niveau universitaire, une entrevue structuree et un questionnaire administre aux candidats. Pour evaluer la connaissance que les postulants pouvaient avoir du programme et de la profession, l'Ecole d'hygiene dentaire de l'Universite Dalhousie a pris la decision, en 1996, de soumettre les candidats a une epreuve ecrite--un questionnaire--plutot qu'a une entrevue, parce que cette derniere exigeait beaucoup de main-d'oeuvre et posait des difficultes d'ordre logistique. Pour chercher a savoir comment les postulants ayant obtenu de faibles resultats dans leurs cours prealables mais un pointage eleve a l'epreuve ecrite s'en sont tires dans leur programme, nous avons compile les resultats a l'epreuve ecrite pour les comparer au rendement indique par la moyenne ponderee cumulative globale obtenue a la fin de la premiere annee du programme. Au cours de leur premiere annee, parmi les etudiants ayant une MRMU elevee, ceux et celles qui avaient eu des resultats moyens ou eleves a l'epreuve ecrite ont obtenu des rendements sensiblement meilleurs que ceux et celles qui avaient obtenu de faibles notes a l'epreuve ecrite (p<0,01). Et parmi les etudiants ayant obtenu une faible MRMU (#72,5 %), ceux et celles qui avaient obtenu des resultats eleves ou faibles a l'epreuve ecrite ont eu un rendement sensiblement meilleur que ceux et celles qui avaient un resultat moyen a l'epreuve ecrite (p<0,01, p<0,05).

Ces resultats donnent a penser que la connaissance prealable de la profession et du programme influe sur le niveau de rendement des etudiants au cours de la premiere annee du programme. Les etudiants qui ont une bonne comprehension de la profession et du programme surmontent tout desavantage cause par de faibles aptitudes aux etudes, comme porte a le croire une MRMU peu elevee. L'epreuve ecrite semble etre un outil satisfaisant qui aide a choisir les postulants a admettre malgre leur faible MRMU et semble constituer aussi une solution de rechange viable a l'entrevue structuree.

BACKGROUND

ADMISSIONS PROCEDURES ARE DESIGNED TO SELECT from a pool of candidates those who are thought to be best suited for the educational program and profession. Primarily, admissions committees review the applicants' prior academic performance. (1,2,3) University-based professional programs such as dentistry, occupational therapy, nursing, medicine, and physiotherapy have also identified characteristics such as communication skills, ethical sensitivity, decision-making skills, and problem-solving ability as important attributes. (1,2,4-8) Until 1996, at the School of Dental Hygiene, Dalhousie University, applicants were selected by considering scores derived from prior university academic performance and a structured interview. Many health professional programs are attempting to broaden the scope of admissions criteria and are investigating alternatives to prior academic performance and an interview. Pereira describes the evolution of admissions criteria for selecting applicants to the problem-based learning (PBL) dental program at University of Southern California. (4) She notes that high academic achievement worked against the success of applicants in the PBL program and that the traditional admissions criteria did not identify candidates with strong interpersonal, communication or decision-making skills, the ability to integrate information, or independent learning skills--all attributes that contribute to a successful problem-based learner. Hoad-Reddick and MacFarlane suggest that suitable candidates for a PBL dental program possess a number of positive attributes such as dexterity, empathy, and communication skills that are not assessed using academic performance or a structured interview. (5) A study conducted by Cunnington and Norman indicates that MCAT (Medical College Admissions Test) scores and GPAs (grade point averages) bear no relationship to clerkship performance and are "insensitive to the kinds of qualities and skills necessary for being a competent and perhaps compassionate physician." (6) Isenburg and Heater provide a list of admissions criteria for entry-level applicants to a master's degree in occupational therapy that includes a high score on a written essay as an effective indicator of the applicant's potential for successfully completing the program. (7) Wilson describes the process for graduate student selection for a master's in nursing program at the University of Missouri-Kansas City. There, the admissions criteria no longer include an interview but require two written essays, one describing the applicant's career development over the past 10 years and the other describing involvement in professional organizations, attendance at conferences, and volunteer efforts. (8) Although the specific tool differs, the consistent trend for admissions committees considering applicants to health profession programs appears to be the investigation of admissions criteria that will highlight those attributes, other than academic performance, deemed to be positive characteristics for the student and graduate of the program.

The purpose of this paper is to describe the development and implementation of the candidate's questionnaire and to explore relationships between CQ scores and performance as indicated by overall grade point averages at the end of the first year of a two-year university-based dental hygiene program. The pass/fail curriculum design in the second year of the program limits the ability to compare admissions criteria to performance at the end of the program. As the candidate's questionnaire score can influence whether or not a candidate with low academic requirements is accepted, the investigators were most interested in the performance of successful candidates with high CQ scores and low required subjects average (RSA).

DEVELOPMENT AND IMPLEMENTATION OF A CANDIDATE'S QUESTIONNAIRE

Traditionally at Dalhousie University's School of Dental Hygiene, a structured interview and prior academic performance were used to select candidates. The structured interview was developed by the Canadian Dental Association and modified for use with dental hygiene applicants. It was intended to explore the applicant's knowledge of the program and of the profession as well as their values. Approximately 80 of the most competitive candidates were invited to a 45-minute interview conducted by teams of two to three persons composed of dental hygiene faculty and senior dental hygiene students. Since the School has only five full-time faculty members, including the Director, it was difficult to staff the interviews and interviewers found the task to be very labour intensive and time consuming.

Other health professions at Dalhousie University including occupational therapy, physiotherapy, and nursing employ a scored questionnaire as a component of their admissions processes. Applicants are typically asked questions designed to explore their knowledge of the profession to which they are applying. In 1996, the Dental Hygiene faculty chose to develop such a "candidate's questionnaire." This was developed from the questions used by the Canadian Dental Association's structured interview form as adapted for the School of Dental Hygiene and from a questionnaire used by the Schools of Physiotherapy and Occupational Therapy at Dalhousie University. It consists of five questions that faculty consider explore the applicant's knowledge of the program and of the profession as well as the applicant's values and beliefs (see table 1).

The essay-style responses are scored, according to the quality of the answer, into one of three categories:

A: above average, awarded 5 points

B: average, awarded 3 points

C: below average, awarded 1 point

A total score for all five questions is calculated, and used in the admissions procedure.

All full- and part-time faculty of the School of Dental Hygiene were asked to provide the answers they would expect from the applicants and these answers were used to develop a scoring key.

Candidate's responses were scored by the five full-time faculty, each scoring the same question on all papers. This scoring method means that the faculty must be calibrated only with themselves, rather than with their colleagues, as would be required if many faculty members were scoring the same question. For the initial use of the questionnaire, each faculty member self-calibrated by reading samples of answers to evaluate the overall quality of the responses before deciding the standard for each category of A, B, or C. In subsequent years, faculty has scored the same question.

With the addition of the CQ score to the admissions criteria, the rank order of applicants changes from that obtained by using academic components alone. Those applicants with high academic requirements are not substantially impacted by this shift. However, applicants with low academic requirements (and therefore closer to the cut-off point for applicants to be admitted) are the most affected by the CQ score. When the academic qualifications are very similar, a high CQ score may shift the applicant's rank into the range of accepted positions.

METHOD

In compliance with department policy, confidential files such as admissions information and academic performance records were drawn from secure (locked) file cabinets and stored in the Dean's Office, Faculty of Dentistry, Dalhousie University, during the data extraction phase of the study. The data collected for this study included applicants' required subjects average (RSA), the candidate's questionnaire scores (CQ) obtained from admissions information, and the corresponding grade point average (GPA) at the end of the first year of the program from academic records of applicants admitted between 1996 and 2002. All data were coded and processed blind so that no individual in the study could be identified individually. Every effort was taken to guarantee the confidentiality of the student records reviewed. The scope of the study is limited to those applicants who were offered and accepted a position in the program and completed at least the first year of study. The study cohort included a sample size of 250 students. The CQ and RSA scores were categorized into three groups, respectively, using the following criteria: A student scoring 14 or lower out of 25 on the CQ scale was classified as "low." Those students with a CQ score of 18 or higher out of 25 or more were classed as "high" on the CQ index. Students with a score between 15 and 17 fell into the "medium" category. Likewise, students were relegated to the "low" RSA group if their RSA was 72.5% or less while students with a score of 80.5% or better were deemed to be "high" RSA. The mid-range category for RSA rested between 72.5% and 83.5% without including these limit. (See table 2.)

All nine of the CQ by RSA crossed-combinations resulted in mutually exclusive groups or complex categories (i.e., one CQ level combined with one RSA level). Thus, nine groups of successful candidates were analyzed in the study. The mean GPAs were calculated for successful candidates who had (1) a low RSA and a low, medium, or high CQ score; or (2) a medium RSA and a low, medium, and high CQ score; or (3) a high RSA and a low, medium, or high CQ score. Table 3 lists the average grade point averages of the nine complex categories of successful candidates.

A two-way fixed effects, analysis of variance (ANOVA) was performed on the data in an effort to evaluate differences on the GPA for (1) the three CQ groups, alone (main effect of CQ); (2) the three RSA groups, alone (main effect of RSA); and (3) most importantly, the interaction between these two factors. In accordance with the simple effects strategy described by Kirk, (9) an ANOVA source table was derived with the intention that significant omnibus F-tests would be followed by appropriate post-hoc analytics (e.g., simple main effects tests and pairwise contrasts, suitably corrected for inherent type I errors). (See table 4.)

RESULTS

Table 3 summarizes the sample sizes (n), means (M), and standard deviations (SD) of the grade point averages for the levels or groups of the two primary factors and the complete set of nine complex interactive subgroups representing the interaction. The mean GPA for all students (n = 250) was 3.30 (SD = 0.40). The 86 successful candidates occupying the "low" CQ category had a mean GPA of 3.27 (SD = 0.36) while the 56 candidates in the "high" CQ group had a mean GPA of 3.38 (SD = 0.37). The remaining candidates (n = 108), comprising the "medium" interval, revealed an average GPA of 3.28 (SD = 0.45). Among the RSA groupings, 63 candidates in the "low" category obtained a mean GPA of 3.21 (SD = 0.44). The 130 students falling into the "medium" RSA group returned an average GPA of 3.29 (SD = 0.40), and the "high" RSA group (n = 57) had a mean GPA of 3.41 (SD = 0.35).

Averaged GPA scores among the nine dual-factor subgroups representing the interaction were varied as well. However, they followed a more involved and less-linear pattern of relative scores than that which was evident in the two main factors of CQ or RSA alone. For instance, the low CQ/high RSA subgroup (n = 14, M = 3.13, SD = 0.24) represented one of the lowest GPA averages; the 19 students occupying the high CQ/low RSA subcategory (n = 19, M = 3.34, SD = 0.39) show one of the higher mean GPAs in the study. The group with the lowest average GPA was the medium CQ/low RSA subgroup (n = 23, M = 3.03, SD = 0.49) while the highest mean score was, not unexpectedly, achieved in the high CQ/high RSA category (n = 13, M = 3.54, SD = 0.31).

As shown in Table 4, the two-way ANOVA source table revealed no significant main effects for either the RSA or CQ groups. However, a significant interaction between the RSA and CQ groups was evident [F (4, 241) = 3.71, p [less than or equal to] 0.01].

Figure 1 illustrates the "RSA by CQ Groups" interaction. A post-hoc simple main effects strategy was used to localize the origins of the significant omnibus test of the interaction reported in table 4. The only one-way simple main effects ANOVA within this triad of tests that proved significant was that of the medium CQ scorers between the three RSA subgroups [F (2, 241) = 8.91, p [less than or equal to] 0.001]. Within this subcategory of RSA levels, a further examination of the pairwise-differences between groups was conducted using simple effects-contrasts analysis. This enabled the investigators to determine the level of statistical significance for the mean differences between an exhaustive set of six pairings of the three RSA subgroups.

Among students with a medium CQ score, those with high or medium RSA standing, performed significantly better in the program than students with a low RSA score (low RSA M = 3.03 < medium RSA M = 3.26, p [less than or equal to] 0.05; low RSA M = 3.03 < high RSA M = 3.49, p [less than or equal to] 0.001). In keeping with the observed linear trend for an increase in GPA commensurate with low to high RSA standing, students performing in the high RSA subgroup scored significantly better than students occupying the corresponding medium RSA subgroup (medium RSA M = 3.26 < high RSA M = 3.49, p [less than or equal to] 0.05).

Figure 2 illustrates the complimentary perspective of the interaction by exploring the CQ by RSA subgroup dynamic. Again, the remaining three possible one-way simple main effects ANOVAs were performed. However, in this set of tests, the RSA categories were held constant while the CQ subgroupings were examined for significance. Two of the three one-way simple main effects tests proved significant at p [less than or equal to] 0.05. Differences among the CQ subgroups were evident for students occupying the low RSA group [F (2, 241) = 4.35, p [less than or equal to] 0.05] and the high RSA group [F (2, 241) = 6.24, p [less than or equal] 0.01]. In keeping with the prescribed analytic strategy, (9) post-hoc simple effects contrasts were performed on the two significant simple effects. With respect to the first-year students in the high RSA category, those with a high or medium CQ score performed significantly better than those with a low CQ score (low CQ M = 3.13 < medium CQ M = 3.49, p [less than or equal to] 0.01; low CQ M = 3.13 < high CQ M = 3.54, p [less than or equal to] 0.01). Interestingly, among students with a low RSA, those with a low or a high CQ score performed significantly better than those with a medium CQ score (medium CQ M = 3.03 < low CQ M = 3.29, p [less than or equal to] 0.05; medium CQ M = 3.03 <high CQ M = 3.34, p [less than or equal to] 0.01). First-year students in the medium RSA group performed similarly regardless of their CQ score.

DISCUSSION

These results suggest the applicant's level of knowledge and understanding of the profession and the program influences the student's performance level in the first year of the program. Students with a better understanding of the profession and the program for which they applied (high CQ) overcome any disadvantage presented by low didactic ability (low RSA). In addition, students who had a poor understanding of the profession and program (low CQ) but who perform at a high level as evidenced by high RSA appear less prepared to meet the challenges presented in the first year of the program and therefore performed at a lower level. Students with a low CQ and a high RSA may represent applicants who apply to a wide range of programs and thus dental hygiene may not be their preferred program.

Faculty members have identified several advantages to using the CQ as an admissions criterion. For example, the time required to administer and score the questionnaire is considerably less than the time required to interview eligible applicants. Interviews required at least two full days of each faculty member's time. In contrast, the questionnaires can be scored in three to four hours. In addition, while the subjectivity of the interview introduces the possibility of bias, this is significantly reduced in the objective CQ. The faculty member scoring the questionnaire knows nothing about the candidate except that she or he was qualified to write the candidate's questionnaire. Finally, because each faculty member scores one question only, the possibility of influence or potential for bias in the CQ score is minimized and intra-rater reliability is high.

Unsuccessful applicants are often counselled regarding actions to take to improve the likelihood of acceptance should they decide to re-apply to the School of Dental Hygiene. In cases where their academic requirements are low, it may be suggested that they take additional university courses to improve their standing. If their academic qualifications are low and they had a low CQ score, it may be suggested that they investigate the profession further as well as improve their academic record.

The potential for re-applicants to complete the questionnaire more than once raised the issue of whether or not the questions on the questionnaire should be different from one year to the next. Faculty reasoned that the re-applicant who answers the questionnaire a second time is likely someone with low academic requirements. An unsuccessful candidate will be aware of the questions asked on the CQ and therefore have the opportunity to become more prepared to answer the questions. Faculty agreed that an improvement in the overall ranking as a result of a better score on the CQ was acceptable because it indicated the applicant was more knowledgeable about their chosen profession.

Because the CQ is intended to identify those applicants who have the best background knowledge of their chosen profession as well as appropriate values and beliefs, faculty raised the question of whether or not there was a CQ score below which the candidate should be denied acceptance regardless of their academic performance. This question is more difficult to answer, but these results seem to suggest that students admitted with low ([less than or equal to]14/25) CQ scores earned on average grade point averages of 3.27 as compared with the 3.38 grade point averages of those with high ([greater than or equal to] 18/25) CQ scores.

SUMMARY

The results suggest that, for admissions committees, the CQ score may be a valuable component of the admissions process. Although the number of students in each subgroup is small, these data indicate that the candidate's questionnaire may be a reasonable alternative to the structured interview in discriminating among applicants with low academic requirements.

Regardless of high, medium, or low RSA or CQ, the applicants selected perform at an acceptable level. Applicants with more knowledge about the profession and program perform better than those applicants who do not have that information. It could be argued that those with high CQ scores are applicants who are more likely to be satisfied with their career choice and remain in the profession. Perhaps admissions committees should consider increasing the emphasis placed on the candidate's questionnaire score when considering those applicants with high required subject averages.

IMPLICATIONS

The School of Dental Hygiene at Dalhousie University found the administration of structured interviews to be logistically difficult and labour intensive. The candidate's questionnaire was designed to explore the same attributes as the structured interview and was favourably received as a part of the admissions process. Anecdotally, the Admissions Committee reported that the accumulated score from the questionnaire provided a discriminating factor to the selection process that the structured interview scores had not. However, as the Committee continues to work with the candidate's questionnaire, several questions have arisen.

The study shows students with high academic requirements are accepted and will perform well even though they may not be prepared for the demands of the program and profession. However, would the student and profession be better served if the admissions committee placed a greater emphasis on the CQ score, thereby selecting candidates who have demonstrated their knowledge of the profession? Further, when considering qualified applicants with low academic requirements, should those with high CQ scores be offered positions before those with high academic requirements and low CQ? Is there a CQ score below which the applicant should be denied a position regardless of their required subjects average?

REFERENCES

1. Hupp JR. Admissions committees select our future dental professionals. Right? Oral Surg Oral Med Oral Path Oral Radiol Endod. 2004;97(1):1-2.

2. Scott AH, Chase LM, Lefkowitz R, Morton-Rias D, Chambers C, Joe J, Holmes G, Bloomberg S. A national survey of admissions criteria and processes in selected allied health professions. J Allied Health. 1995;24(2):95-107.

3. Mitchell TL, MacInnis WA, Murphy HJ. Predictors of performance in dental hygiene education. Probe. 1988;22(1):35-37.

4. Pereira LS. Admission Processes in the Dental PBL at USC. J Dent Educ. 1998;62(9):680-84.

5. Hoad-Reddick G, Macfarlane TV. An analysis of an admissions system: can performance in the first year of the dental course be predicted? Br Dent J. 1999;186(3):138-42.

6. Cunnington JPW, Norman GR. Obstacles to identifying valid performance predictors. Acad Med. 1997;72(11):931-32.

7. Isenburg BD, Heater SL. Professionalization of the field: how can educational programs identify the best applicants? Amer J Occup Ther. 1994;48(8):758-59.

8. Wilson T. A student selection method and predictors of success in a graduate nursing program. J Nurs Educ. 1999;38(4):183-87.

9. Kirk RE. Experimental design: procedures for the behavioral sciences. 2nd ed. Monterey (CA): Brooks-Cole, 1982. p. 365-71.

by Terry L. Mitchell, BSc, DDH, MEd, CGN;* D. Wayne Dunham, MA;** and H. Joseph Murphy, EdD[dagger]

* School of Dental Hygiene, Dalhousie University

** Varimax Statistical Research, Ottawa

[dagger] Department of Dental Clinical Sciences, Dalhousie University

1a. What do you know about the profession of dental hygiene, and why did
    you choose to apply?
1b. What are some of the concerns or challenges you have about working
    as a dental hygienist?
2. Imagine that you are a graduate dental hygienist. What do you feel
   your patient has a right to expect from the dental hygiene care that
   you provide?
3. What characteristics and behaviors can be expected of you as a
   student in a professional program?
4. Describe how your work activities, volunteer work, community service,
   and hobbies have prepared you for a career in dental hygiene.
5. The following is a hypothetical question. Read carefully. There is no
   right or wrong answer.

   Suppose you are in a lab course and a close friend is your lab
   partner. You know that your friend is having serious personal
   problems and as a result is having trouble in university. Your
   friend has not done any lab work and you have completed the required
   work. What would you do if your friend asks to copy your
   assignments? Explain your reasoning.

Table 1. The candidate's questionnaire

CQ Groups

High  ([greater than or equal to]18/25)
Mid   (15-17/25)
Low   ([less than or equal to]14/25)

RSA Groups

High  ([greater than or equal to]80.5%)
Mid   (72.6-80.4%)
Low   ([less than or equal to]72.5%)

Table 2. Required subjects average groups

RSA     CQ             Std.
group   group   Mean   deviation  N

Low     Low     3.291  .369        21
        Medium  3.030  .494        23
        High    3.341  .387        19
        Total   3.211  .440        63
Medium  Low     3.306  .383        51
        Medium  3.263  .429        55
        High    3.317  .368        24
        Total   3.290  .398       130
High    Low     3.130  .242        14
        Medium  3.489  .336        30
        High    3.538  .305        13
        Total   3.412  .345        57
Total   Low     3.273  .362        86
        Medium  3.276  .447       108
        High    3.376  .366        56
        Total   3.298  .403       250

Note: Dependent variable: year-1 grade point average

Table 3. Mean year-1 grade point averages (GPA) of admitted students by
admissions standings on candidate's questionnaire (CQ) and required
subjects average (RSA)

Source of   Sum of         Mean
Variation   squares   df   square  F      Sig.

RSA Group       .754    2  .377    2.501   .084
CQ Group        .849    2  .425    2.815  0.62
RSA Group*     2.238    4  .560    3.711   .006
CQ Group
Error         36.343  241  .151
Total       2759.285  250

Note: Dependent variable: year-1 grade point average

Table 4. ANOVA source table: tests of between-subjects effects