Printer Friendly

Predictive validity of medical and dental colleges' entrance test in Punjab: The way forward.

Byline: Junaid Sarfraz Khan Osama Mukhtar Saima Tabasum

Abstract

Objective: To identify the strengths and weaknesses of the current selection process for admissions to public medical colleges being followed in the province of Punjab and to suggest changes.

Methods: The prospective study comprised candidates sitting the Medical and Dental Colleges Entrance Tests administered by the University of Health Sciences Lahore from 2008 to 2011. The marks of pre- and post-admission examinations were entered in SPSS 17 and performance of candidates in different examinations up to 2012 was compared using parametric statistical tests. Pless than 0.05 was considered statistically significant. Reliability of the entrance test question paper was calculated through Cronbach's Alpha.

Results: Overall 101273 candidates sat in the entrance test during the study period and of them 14 995 (14.8%) students were admitted to public and private medical colleges in Punjab affiliated with the University of Health Sciences Lahore. Cronbach's Alpha showed excellent internal consistency reliability of the question paper of 220 items for all these four years. Overall students scored significantly lower marks (pless than 0.05) in all their MBBS Professional Examinations than their pre-admission tests. A similar pattern of difference was seen in entrance test marks in different geographical localities as in the case with MBBS Professional Examination. Students from Islamabad and Rawalpindi regions scored significantly higher marks in all MBBS examinations as well as in the entrance test and aptitude test but lower in the Higher Secondary School Certificate examination.

Internal consistency reliability of the aptitude test was found to be 0.83. Linear regression analysis showed that Estimated Regression Coefficient of the Higher Secondary School Certificate marks was negative but positive for entrance test and indicated that the latter results were a good positive predictor for the post-selection performance. In the sub-components only the Biology and Chemistry regression coefficient section was found to be positive.

Conclusions: Entrance and aptitude tests in Punjab were found to be valid and reliable with incremental predictive validity at least for performance in the pre-clinical years. It is suggested that Higher Secondary School Certificate - marks should have little if any weightage in the selection process.

Keywords: Medical and Dental Colleges Admission Test Aptitude test Health Professional education Selection process Predictive validity. (JPMA 64: 1132; 2014)

Introduction

Professional health educational institutions in Pakistan are faced with the dilemma of having a limited number of seats against a disproportionately high number of potential candidates all seemingly suitable for admission into medical and dental colleges. In Pakistan medical and dental colleges can be divided into two categories " private and public. Public medical colleges are under the direct control of provincial governments and professional health education in these institutions is subsidised and students pay a very nominal tuition fee. Both public and private medical colleges require accreditation licensure and certification by the provincial governments and the Pakistan Medical and Dental Council (PMDC) which is the regulatory authority for medical education in the country as well as by respective degree-awarding universities.

In the province of Punjab currently there are 14 public and 18 private medical colleges with a total number of 3305 seats of which 2980 are in the public sector.1 For admissions in both private and public medical colleges candidates who have scored over 60% marks in the Higher Secondary School Certificate (HSSC) examination are required to sit in the Entrance Test (ET) organised by the respective provincial government in the province of Punjab which has a population of over 80 million.2 The ET is conducted for the provincial government by the University of Health Sciences (UHS) Lahore in September each year. The ET consists of 220 multiple choice questions (MCQs) of single best type with four options and the paper is divided into four sections with 88 questions in Biology 44 in Physics 58 in Chemistry and 30 in English.3 Each correct answer by a candidate carries 5 marks and each incorrect answer a -1 mark. The duration of the exam is two-and-a-half hours.

There is also a non- weighted additional half-an-hour test termed the 'Aptitude and Feedback' test (AT) comprising 60 questions of which 8 are related to feedback regarding the conduct of the ET 30 are related to personality assessment and the rest to aptitude assessment with a mix of questions on ethics critical thinking and abstract non-verbal reasoning.

The problem of mismatch between the available numbers of suitable candidates is not unique to Pakistan. In fact the situation has been the same all over the world for many decades.45 To justify selection various countries have adopted different strategies.6-12 For example in the Netherlands the selection is based partly on institutional selection procedures (up to 50%) and partly on a national lottery weighted for school performance.4 In Australia they use three components in the selection process; the pre-medical academic score the Undergraduate Medicine and Health Sciences Admission Test (UMAT ) and a selection interview.5 In the United Kingdom clinical aptitude test assessing cognitive ability in verbal reasoning quantitative reasoning abstract reasoning and decision analysis is part of the selection process which may include interviews pre-medical achievement scores and letters of reference.1314

The purpose of the selection procedures adopted in the developed countries is to assess cognitive abilities which reflect intellectual capabilities factual recall and performances of test on declarative knowledge while another component commonly known as the non- cognitive component is gaining popularity in the selection process and includes personal qualities attributes attitudes interests and values.14 Of these good communication skills have been found to be an important attribute for both medical students and doctors whereas other cognitive attributes and pre-medical achievement scores are good predictors for performance in the early non-clinical years of medical education.5 Similarly personality traits like openness agreeableness emotional stability and extroversion have been found to be good predictors of better performance in the clinical years of medical education and in professional practice.14

The province of Punjab is vast and potential candidates come from varying socio-economic and educational backgrounds. Pre-medical academic achievement amongst the candidates in the Punjab has been found to be unreliable in predicting future performance of medical students.15 Pakistan is a developing country which requires rationalisation of resource allocation in all areas of governance healthcare education etc. For Pakistan and its population it is therefore important to choose the best possible candidates for admission in professional health education in order to maximise the impact of professionals on the healthcare delivery system.15

The aim of this study was to identify the strengths and shortcomings of the current selection process for admissions in public medical colleges being followed in Punjab and to suggest changes in order to improve the quality of the selection based on the Best Evidence Medical Education (BEME) practices worldwide.

Material and Methods

The prospective study comprised candidates sitting the Medical and Dental Colleges Entrance Tests administered by UHS Lahore from 2008 to 2011. Reliability of ET question paper was calculated using Cronbach's Alpha. Students who qualified for admission in medical colleges affiliated with UHS all over Punjab following the ET were the focus of the study. The marks of pre-admission examinations Secondary School Certificate (SSC) HSSC ET and AT and post-admission examinations (MBBS Professional Examinations) were entered into SPSS 17. The pre- and post-admission performances of these candidates in different examinations up to 2012 were compared using paired sample t-test. Marks of all MBBS Professional Examinations and SSC marks were adjusted to make the total equivalent to 1100 as per the ET and HSSC marks. Analysis of Variance (ANOVA) was used to compare the performances of candidates from different geographic localities.

The whole cohort was distributed in 11 different geographical localities; Nine (09) regions (Divisions) of Punjab which consisted of 36 districts and two others were Islamabad and Azad Jammu and Kashmir. There were 6 districts that were considered underdeveloped i.e. D.G. Khan Muzaffargargh Rajanpur (included in D.G. Khan division) Layyah (included in Multan) and Bhakkar (included in Sargodha). Regression analysis was used to determine the relationship between the performance of the students in MBBS Professional Examinations and their pre-admission examinations. Pless than 0.05) was considered statistically significant.

Results

Overall 101273 candidates appeared in the ET from 2008 to 2011 and of these 14 995 (14.8%) students were admitted in all public and private medical colleges in the Punjab affiliated with the UHS. Year-wise admitted students are presented in Table-1.

The Cronbach's Alpha showed excellent internal consistency reliability of the ET question paper of 220 items for all these four years. Cronbach's Alpha for each section of ET was found to be reasonably high. Results of

Table-1: Year-wise admitted students.

Year###Number of###Number of###Percentage

###Candidates appeared###Candidates###of Admitted

###in Entrance Test###Admitted###Candidates

2008###18270###2943###19.63%###

2009###20231###3168###21.13%

2010###30026###4218###28.13%

2011###32746###4666###31.12%

Total###101273###14995###100.00%

Table-2: Results of internal consistency reliability (Cronbach's Alpha).

Medical and Dental###Cronbach's Alpha###

Admission Tests###(All Items) (Physics) (Chemistry) (Biology) (English)

2011###0.97###0.86###0.94###0.95###0.76

2010###0.96###0.90###0.89###0.90###0.73

2009###0.96###0.83###0.94###0.91###0.81

2008###0.96###0.81###0.79###0.88###0.78

Paired Sample T test showed that overall students scored significantly lower marks (pless than 0.05) in their MBBS Professional Examinations (adjusted) than their pre- admission examinations. Results are presented in Table-3.

Analysis of Variance (ANOVA) showed a significant difference in the mean marks of medical students belonging to different geographical localities in the pre- and post- admission examination in all cohorts of the study subjects (pless than 0.05) except for some variation in the 2011 cohort (p greater than 0.05). ANOVA results are shown in Table-4.

ANOVA results of ET from 2008 to 2010 showed almost similar pattern of difference in ET marks in different localities as that of all MBBS Professional Examination results. Students from Islamabad and Rawalpindi region scored significantly higher marks in the MBBS Professional Examinations as well as in the ET. The pattern was reversed when compared with the HSSC results. Students from D.G. Khan region scored least in all MBBS Professional Examinations but scored highest in their HSSC examination. Results are presented in Figure-1.

For the cohort of 2011 two more pre-admission examination results (Adjusted Aptitude and Matriculation (SC) results) were included. Internal consistency reliability (Cronbach's Alpha) value of the Aptitude Test was found to be 0.83 which shows that this tool of assessment is reliable. The pattern of difference in MBBS 1st Professional Part I (P-I) is similar to the Entrance and Aptitude tests. The candidates belonging to developed districts scored higher in the MBBS 1st Professional P-I as was the case in the Entrance and Aptitude test results. The pattern of difference in the MBBS 1st Professional P-I results of different localities doesn't match the SSC and HSSC results. Results are presented in Figure-2.

Linear regression analysis calculated the following regression equations for the MBBS professional examinations by taking Entrance Test and HSSC marks as predictive variables in first step. Results are presented in Table-5.

Table-3: Comparison of Entrance Test and HSSC marks with MBBS Professional Examinations (Adjusted).

###Paired Sample T-test###MeanS.D###T-Statistics###P-Value

Pair 1###Entrance Test Marks###818.6122.0###37.0###0.00

###Adjusted MBBS First Professional Part-I###734.194.1

Pair 2###Entrance Test Marks###813.5126.2###29.1###0.00

###Adjusted MBBS First Professional Part-II###751.567.3

Pair 3###Entrance Test Marks###819.6128.4###28.0###0.00

###Adjusted MBBS Second Professional###750.749.2

Pair 4###Entrance Test Marks###848.9106.4###36.1###0.00

###Adjusted MBBS Third Professional###745.752.0

Pair 5###HSSC Marks###927.547.9###118.9###0.00

###Adjusted MBBS First Professional Part-I###734.194.1

Pair 6###HSSC Marks###925.449.3###135.0###0.00

###Adjusted MBBS First Professional Part-II###751.467.3

Pair 7###HSSC Marks###926.447.4###145.1###0.00

###Adjusted MBBS Second Professional###750.749.2

Pair 8###HSSC Marks###919.144.3###101.5###0.00

###Adjusted MBBS Third Professional###745.752.0

Table-4: Comparison of the Performance of Medical Students coming from different geographical localities in pre- and post-admission Examinations.

Examinations (Marks)###OVERALL###COHORTS

###2008###2009###2010###2011

###F-Statistics###p###F-Statistics###p###F-Statistics###p###F-Statistics###p###F-Statistics###p

Adjusted Aptitude Marks###2.06###0.03###-###-###-###-###-###-###2.06###0.03

Adjusted Matric Marks###1.08###0.37###-###-###-###-###-###-###1.08###0.37

Entrance Test Marks###13.93###0.00###5.05###0.00###6.87###0.00###6.53###0.00###1.08###0.38

HSSC Marks###25.20###0.00###9.82###0.00###15.39###0.00###3.90###0.00###2.88###0.00

MBBS First Professional Part-I###13.63###0.00###9.65###0.00###9.57###0.00###3.49###0.00###0.90###0.53

MBBS First Professional Part-II###14.03###0.00###7.72###0.00###8.19###0.00###1.70###0.00###-###-

MBBS Second Professional###8.79###0.00###3.61###0.00###6.34###0.00###-###-###-###-

MBBS Third Professional###14.18###0.00###14.18###0.00###-###-###-###-###-###-

Table-5: Regression Equations for different MBBS professional examinations.

Level/Step###Dependent Variable###Independent/ Predictor Variable###Linear Regression Equation

Pre Admission Examination###MBBS 1st Prof. Part-I###Entrance Test and HSSC marks###787.43 + 0.114 E.T - 0.141 HSSC

###MBBS 1st Prof. Part-II###Entrance Test and HSSC marks###781.96 + 0.083 E.T - 0.106 HSSC

###MBBS 2nd Professional###Entrance Test and HSSC marks###702.43 + 0.012 E.T - 0.042 HSSC

###MBBS 3rd Professional###Entrance Test and HSSC marks###772.298 + 0.072 E.T - 0.095 HSSC

Components (sections) of the Entrance Test MBBS 1st Prof. Part-I###Biology Chemistry Physics and English Marks###347.586+0.175 (Bio)+0.801 (Chem) - 0.579 (Phy) - 0.364 (Eng)

###MBBS 1st Prof. Part-II###Biology Chemistry Physics and English Marks###343.326+0.017 (Bio)+0.633 (Chem) - 0.166 (Phy) - 0.382 (Eng)

###MBBS 2nd Professional###Biology Chemistry Physics and English Marks###358.923+0.210 (Bio)+0.512 (Chem) - 0.073 (Phy) - 0.128 (Eng)

###MBBS 3rd Professional###Biology Chemistry Physics and English Marks###367.321+0.094 (Bio)+0.631 (Chem) - 0.127 (Phy) - 0.028 (Eng)

All equations in the first step of pre-admission examinations show that Estimated Regression coefficient of HSSC marks is negative but positive for Entrance test (ET ) marks which predict that the value of MBBS Professional Examination decreases by the respective units above for every unit increase in HSSC marks when other variables are held constant in each equation. But the value of MBBS Professional examination increases by the respective units above for every unit increase in Entrance test marks when other variables are kept constant in each equation.

The Regression Analysis indicates that ET results are a good positive predictor for the performance of students in their MBBS professional examinations.

In second step regression analysis was applied on MBBS Professional Examination performances by taking four components (Sections) of the Entrance Test as independent variables. Results are presented in Table-4.

Estimated Regression Coefficient for Biology and Chemistry section was found to be positive in all these equations which predict that value of MBBS Professional Examinations increase as indicated with a unit increase in Biology and Chemistry section marks by taking all other variables constant but decreases in the case of Physics and English. Discussion

Selection processes procedures protocols methodologies etc adopted over the years in various countries and regions have not been without their controversies.316-18 These controversies arise as a result of the high political societal and psychological stakes involved in conducting admission or selection tests. In Pakistan in the province of Punjab around 36000 candidates secure over 60% marks in the HSSC examination. The seats available for admissions in public medical colleges are limited to only 2980 so all these 36000 candidates cannot be admitted. A simple lottery weighted for academic achievement as is the case in the Netherlands is difficult to justify when faced with this huge volume of applicants. The same holds for other auxiliary methods of selection like letters of references or intent and semi-structured mini-interviews.

The authors believe that interviews alone for 36000 candidates will not be cost-effective and will be beset with the problems of lack of standardization inter-rater bias gender and demographic bias and political influences. With reported evidence of poor-reliability of pre-medical academic achievement for predicting future performance in health professions education this variable alone also cannot be the sole determining factor for selection. ET for selection of candidates for admission in medical colleges in Punjab has been in place since 1998. The Government of Punjab handed over the conduct of ET to the University of Health Sciences Lahore (UHS) in 2008. And to date the UHS has conducted five regular ETs for the province of Punjab for admission of candidates in health professions education. The syllabus for 2010 was revised to reflect the distribution presented in the introduction section together with introduction of a non-weighted 'Aptitude and Feedback' test (AT).

Since the population for the test comes from varied educational backgrounds including both the Punjab HSSC Board education and other educational systems national as well as foreign like for example Advanced Level of UK (A-level) a syllabus was developed by the UHS with learning objectives derived from commonalities in the various syllabi. The syllabus and the table of specifications derived from the syllabus reflect the crystallized intelligence derived from the premedical education streams. This the authors believe establishes the construct validity of entrance test since all questions asked in ET can be cross-referenced from the textbooks and syllabi of both the Punjab Higher Secondary School Board and the non-board based educational streams. The reliability co-efficient presented in the result section of the paper for the ET and its sub- components indicate a high internal consistency for this test over the years. With the construct validity and reliability established the authors would now like to focus their attention on predictive validity. The results clearly indicate that Entrance test has high incremental predictive validity with better performance in Entrance test predicting higher scores in health professions education. This is not the case when comparing pre-ET academic achievement with post-medical- selection academic performance. This incremental predictive validity is at least established for the pre-clinical years of health profession education. Only time will prove the worth of ET in predicting educational performance in the clinical years and later on in professional life.

The authors believe that by combining the data across the years they have presented a more realistic picture of predictive validity. However as is evident from Table-2 and Figures 1 and 2 and the data from different years presented separately there is high positive correlation between Entrance Test scores and post-selection educational performance when compared with the correlations between HSSC and SC performance and the same post-selection medical college performance.

From the result it is also evident that all four components of Entrance test do not have similar Predictive validity for post-selection performance. However it is not the intent of the authors to suggest that one or more components may be excluded from the Entrance test. Nevertheless the evidence in support of excluding English and Physics sub- components from the ET in the future is substantial and these portions in the opinion of the authors can be replaced with the aptitude test discussed next in this section.

In discussing the 'Aptitude and Feedback' test the authors would like to add that in this study the data presented is only for the first 22 out of 60 questions. These 22 questions evaluate ethical and behavioral concepts abstract non-verbal reasoning and aptitude and hence are a mix of both crystallized intelligence and fluid intelligence.3 0.5 as a value for Crohnbach's alpha is acceptable given the short version of this test (22 items).Of greater importance is the fact that the data from 2011 suggests incremental predictive validity comparable to that of the Entrance Test for post-selection academic achievement. This is particularly exciting for the authors believe that over the years further data shall be available to match predictive validity of Entrance Test and the aptitude test against each other in predicting not only pre-clinical academic performance but also clinical academic performance and clinical practice in professional life. The data suggests bias related to social class and educational background. Surprisingly students from socio-economically and educationally under-developed districts (DG Khan Lodhran Muzaffargarh Rajanpur Layyah) are seen consistently to score higher marks in SC and HSSC examinations than in post-medical selection professional examinations. Since it is the performance after selection that is the subject of interest here the authors believe that Entrance and Aptitude tests being better predictors of post selection academic performance should be given greater or sole weightage in the selection process. Whereas it is not the purpose of this study to comment on the reasons for this poor predictive validity of School Certificate (SC) and HSSC results the authors would like to add only that better performance by candidates from underdeveloped districts in SC and HSSC examinations in their own districts may be a result of leniency on the part of the individual

Districts Boards of underdeveloped districts trying to follow a policy of widening participation unofficially.

Conclusions

Based on the results of the study the authors conclude that Entrance test conducted by UHS Lahore for Government of Punjab for admissions in medical colleges in the province is a valid and reliable test with incremental predictive validity at least for performance in the pre- clinical years of medical education so far. However the authors would like to suggest that in order to reduce the economic and psychosocial burden of the test the eligibility for applicants to sit in the test might be raised from 60% HSSC marks to 70% thus reducing the test burden. Under the current circumstances in which HSSC and SC scores have poor predictive validity whereas Entrance and Aptitude tests have high predictive validity at least for performance in the pre-clinical years it is suggested that HSSC and SC marks should have little if any weightage in the selection process.

References

1. Government of the Punjab. Prospectus for admissions to Government Medical and Dental Institutions of the Punjab session 2011-2012. University of health Sciences. (Online) (Cited 2013 May 24); Available from URL: http://www.uhs.edu.pk/mcat/full_prospectus_final.pdf 2. Pakistan Medical and Dental Council: Conditions for admission in MBBS/BDS courses and conditions for house job/internship/foundation. (Online) 2013 (Cited 2013 May 25); Available from URL: http://pmdc.org.pk/portals/0/admission% 20housejob%20regulations.pdf

3. University of Health Sciences. Syllabus for Entrance Test 2013 University of Health Sciences. (Online) 2013 (Cited 2013 May 24); Available from URL: http://www.uhs.edu.pk/mcat/etsyllabus2013.pdf

4. Urlings-Strop LC Stegers-Jager KM Stijnen T Themmen AP. Academic and non-academic selection criteria in predicting medical school performance. Med Teach 2013; 35: 497-502.

5. Mercer A Puddey IB. Admission selection criteria as predictors of outcomes in an undergraduate medical course: A prospective study. Med Teach 2011; 33: 997-1004.

6. Mcmanus IC Ferguson E Wakeford R Powis D James D. Predictive validity of the Biomedical Admissions Test: An evaluation and case study. Med Teach 2011; 33: 53-7.

7. Ferguson E James D Madelely L. Factors associated with success in medical school: Systematic review of the literature. BMJ 2002; 324: 952-7.

8. Parry JM Mathers JM Stevens AJ Parsons A Lilford R Spurgeon P et al. Admissions processes for five year medical courses at English schools: review. BMJ 2006; 332: 1005-9.

9. Kulatunga-Moruzi C Norman GR. Validity of admissions measures in predicting performance outcomes: The contribution of cognitive and non-cognitive dimensions. Teach Learn Med 2002; 14: 32-42.

10. Siu E Reiter HI. Overview: What's worked and what hasn't as a guide towards predictive admissions tool development. Adv Health Sci Educ 2009; 14: 759-75.

11. Patterson F Ferguson E. Selection for medical education and training. In: Swanwick T editor. Understanding medical education. London UK: Wiley-Blackwell 2010.

12. Prideaux D Roberts C Eva K Centeno A Mc Corie P Mc Manus C et al.. Assessment for selection for the health care professions and specialty training: conference. Med Teach 2011; 33: 215-23.

13. Turner R Nicholson S. Can the UK Clinical Aptitude Test (UKCAT) select suitable candidates for interview Med Educ 2011; 45: 1041-7.

14. Hojat M Erdmann JB Gonnella JS. Personality assessments and outcomes in medical education and the practice of medicine: AMEE Guide no. 79. Med Teach 2013; 1-35.

15. Khan JS Tabasum S Mukhtar O. Comparison of pre-medical academic achievement entrance test and aptitude test scores in admission selection process. J Pak Med Assoc 2013; 63: 552-7.

16. Harden RM. If it matters it produces controversy. Med Teach 2011; 33: 4-5.

17. Mcmanus IC Ferguson E Wakeford R Powis D James D. Response to comments by Emery and Bell Medical Teacher 33(1): (this issue). Med Teach 2011; 33: 60-1.

18. Emery JL Bell JF. Comment on I.C. McManus Eamonn Ferguson Richard Wakeford David Powis and David James (2011). Predictive Validity of the BioMedical Admissions Test (BMAT): An evaluation and case study. Medical Teacher 33(1): (this issue). Med Teach 2011; 33: 58-9.
COPYRIGHT 2014 Asianet-Pakistan
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2014 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Publication:Journal of Pakistan Medical Association
Article Type:Clinical report
Geographic Code:9PAKI
Date:Oct 31, 2014
Words:4240
Previous Article:Effects of hepatitis C virus gene NS2 on the expressions of Bcl-2 and Bax in HepG2 cells.
Next Article:Frequency of severe vitamin-D deficiency in patients presenting to a tertiary care hospital in Islamabad.
Topics:

Terms of use | Privacy policy | Copyright © 2020 Farlex, Inc. | Feedback | For webmasters