Printer Friendly

Establishing construct validity of AMEET (assessment of medical educational environment by the teachers) inventory.

Byline: Rizwana Shahid, Rehan Ahmed Khan and Rahila Yasmeen

Abstract

Objective: To establish the construct validity of Assessment of Medical Educational Environment by the Teachers inventory.

Methods: The cross-sectional analytical study was conducted from January to May 2017 and comprised doctors working as faculty in Rawalpindi Medical College, Rawalpindi, Pakistan, and its 3 teaching hospitals. Non-probability (purposive) sampling was used to meet the criteria of 5 participants per item of the Assessment of Medical Educational Environment by the Teachers inventory. Exploratory factor analysis was done using SPSS 20 and confirmatory factor analysis was done with version 16 of the Analysis of Moment Structures software.

Results: Of the 250 subjects, 126(50.4%) were males and 124(49.6%) were females. Exploratory factor analysis ended with extraction of 11 components. It showed sufficiency of sample size and no multi-collinearity. Three (50%) of the six domains were finalised on the whole and 10(20%) of the 50 items were debarred from the inventory. All three domains had high reliability. Root mean square residual and chi square/degree of freedom were within acceptable limit. However, comparative fit index, goodness of fit index, normed fit index and root mean square error of approximation portrayed not only poor model fit after re-running confirmatory factor analysis, but also led to omission of further 16(32%) items with poor loadings from the inventory. Thus, there was exclusion of total 26(52%) items from the tool and the finalised Assessment of Medical Educational Environment by the Teachers inventory comprised 24(48%) items.

Conclusion: Construct validity of Assessment of Medical Educational Environment by the Teachers inventory could not be established, but the tool was found to be reliable.

Keywords: AMEET inventory, Exploratory factor analysis, Confirmatory factor analysis, Construct validity. (JPMA 69: 34; 2019)

Introduction

Diverse teaching and learning strategies are employed in medical institutes all over the world to enhance learning of the students.1 Communication gap between teachers and students was also found to be adversely affecting the learning climate of medical students. Therefore the need was felt to arouse interest among medical teachers to work for the improvement of the educational climate.2 Numerous instruments were devised to assess educational climate by getting viewpoints of the students like Postgraduate Hospital Educational Environment Measure (PHEEM), Dundee Ready Educational Environment Measure (DREEM) etc., but the Assessment of Medical Educational Environment by the Teachers (AMEET) inventory is a tool that is actually designed to get judgments of the tutors pertinent to educational environment of medical students.3 AMEET is an instrument that is currently used to gauge educational climate of medical students by gathering opinion of the teachers.

This tool is designed by medical teachers of the United Arab Emirates. It is based on 50 items and six domains. This inventory is grounded on a five-point Likert scale from 'strongly agree' to 'strongly disagree'. This tool has excellent reliability (Cronbach alpha 0.94) but unluckily the inventors could not establish its construct validity due to insufficient sample.4 Construct validity can be established by factor analysis keeping in view five respondents per item.5 Construct validity is the extent to which any tool or trial measures what it is supposed to gauge.6 Construct validity is broad-ranged. It covers all the verifications supporting specific interpretation of a score.7 The validation process embraces accretion of proofs in order to have methodical explanation. Establishing construct validity of any tool is a strenuous practice and this could be done through factor analysis both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA).8

We are confronted with diverse terminologies during EFA, while CFA results in the creation of a model and calculation of certain indices.9 The current study was planned to ascertain the construct validity of AMEET inventory in relevance to Pakistani culture and context following its factor analysis. The study was likely to confirm applicability of AMEET inventory in our setup for appraising the learning climate of our medical institutions.

Subjects and Methods

The cross-sectional analytical study was conducted from January to May 2017 and comprised doctors working as faculty in Rawalpindi Medical College (RMC), Rawalpindi, Pakistan, and its 3 teaching hospitals: Holy Family Hospital (HFH), Benazir Bhutto Hospital (BBH) and District Head Quarters (DHQ) Hospital. For factor analysis, data has to be collected from 5 participants per item.5 Non-probability (purposive) sampling was used to meet the criteria for the 50-item inventory. The subjects filled AMEET inventory (Annexure-1) after first giving informed consent. Permission for use of AMEET inventory for the establishment of its construct validity was taken from one of the principal inventors of this instrument.4 Approval was obtained from the ethics review board of Riphah International University, Rawalpindi and RMC. All doctors working in RMC and the three teaching hospitals were enrolled. House officers and postgraduate trainees were excluded. EFA was conducted using SPSS 20.

Items of the AMEET inventory found with Eigen value less than 1 after EFA were eliminated from the inventory. It is mandatory for each domain to have at least 3-5 items for accuracy of the results from EFA.10 CFA was carried out by using version 16 of the Analysis of Moment Structures (AMOS) software. Parameters of CFA were also considered before eliminating irrelevant items. In AMOS 16, visual paths were drawn on graphic windows.

Table-1: Designation of study participants.

Sr.####Designation###Name of Hospital / Institute

###*RMC###*HFH###*BBH###*DHQH

1.###Professor###4###3###7###2

2.###Associate Professor###2###2###1###0

3.###Assistant Professor###8###18###17###6

4.###Senior Registrar###0###35###18###13

5.###Medical Officer###0###16###16###16

6.###Senior demonstrator###25###0###0###0

7.###*APMO/ *APWMO/*PWMO###12###0###0###1

8.###Demonstrator###23###0###0###0

9.###Consultant###0###1###0###4

###Total###74###75###59###42

Table-2: Kaiser-Meyer-Olkin Measure (KMO) and Bartlett's Test.

Kaiser-Meyer-Olkin Measure of Sampling Adequacy.###0.900

Bartlett's Test of Sphericity###Approx. Chi-Square###9603.163

###df###1225

###Sig.###0.000

Table-3: Total Variance.

Sr.#. Components###Initial Eigen Values

###Total###*% of Variance###Cumulative %

1###1###16.89###33.77###33.77

2###2###3.98###7.96###41.74

3###3###3.29###6.59###48.33

4###4###2.39###4.79###53.13

5###5###1.94###3.87###57.00

6###6###1.63###3.27###60.27

7###7###1.37###2.74###63.01

8###8###1.25###2.49###65.51

9###9###1.12###2.24###67.75

10###10###1.06###2.13###69.88

11###11###1.02###2.04###71.92

Table-4: Rotated Component Matrix.

Item No.###Components / Domains

###1###2###3###4###5###6###7###8###9###10###11

28###.817

10###.775

48###.739

21###.733

23###.718

02###.714

11###.703

09###.702

20###.682

12###.670

25###-.666

27###.659

50###-.596###.401

06###.595

19###-.582###.566

08###-.567

30###.565###.416

39###.551###.457

04###.432###-.421

16###.861

17###.814

18###.694

15###-.413###.655

07###.534###.564

14###.510###-.441

35###.508###.453

47###-.738

42###.657

44###-.416###-.538

49###.479###.494

03

43###.774

46###.659

45###.610###.472

01

29###-.779

22###.536###.555

31###.771

32###.766

26###.422

34###.678

33###.567

36###-.471

41###.493###.716

40###.406###.631

05###-.787

24###.704

38

13###.456###.640

37###.435###.446

Table-5: Domains established following exploratory factor analysis (EFA).

Domain / Factor No. 1###Domain / Factor No. 2###Domain / Factor No. 3

Teachers' perception of###Teachers' perception of###Teachers' professional

teaching and learning###learning activities and###self-perceptions

###atmosphere###collaborative atmosphere

###Item 2###Item 7###Item 33

###Item 4###Item 13###Item 37

###Item 6###Item 14###Item 40

###Item 9###Item 15###Item 41

###Item 10###Item 16###Item 42

###Item 11###Item 17###Item 43

###Item 12###Item 18###Item 45

###Item 20###Item 19###Item 46

###Item 21###Item 31###Item 49

###Item 22###Item 32

###Item 23###Item 34

###Item 24###Item 35

###Item 26###Item 50

###Item 27

###Item 28

###Item 30

###Item 39

###Item 48

Table-6: Fit Indices of both models of CFA in comparison with Yardstick.

Fit indices###Indices calculated from initial model###Indices calculated from final model###Yardsticks (Leach, 2008)

*X2/df###4.7###4.7###<5

*RMR###0.08###0.06###0.93

*GFI###0.56###0.70###>0.93

*NFI###0.56###0.74###>0.93

*RMSEA###0.12###0.12###<0.08

Results

Of the 250 subjects, 126(50.4%) were males and 124(49.6%) were females. Institutions and designations of the subjects were noted (Table-1). Kaiser-Meyer-Olkin (KMO) measure of sample adequacy was computed to be 0.900 which indicated sufficient sample size. In addition, highly significant Bartlett's test of sphericity value (0.000) suggested rejection of null hypothesis and efficient application of EFA on the data set with varimax rotation (Table-2). The computed determinant equivalent was 9.6, revealing no multi-collinearity. Percentage of variance and eigen value constituted by each domain was separately (Table-3). Also, 11 components/domains were extracted from AMEET inventory following EFA and varimax data rotation (Table-4). Ten (20%) items were totally eliminated from the inventory after EFA, out of which 8(16%) were excluded because of their negative scores and 2(4%) others because they did not load in any component.

Out of six domains, 3(50%) were finalised and the items of other domains (with less than 3 items) were also shifted to the 3 major domains keeping in view their theoretical confirmation to the respective domain. Ultimately CFA also led to the elimination of further 16(32%) items which were computed to have poor loading (<0.4) in their respective domains (Annexure-2). The 3 major domains were established from results of EFA keeping in view rotated component matrix and reliability (internal consistency) of all the items within each domain (Table-5). EFA was followed by CFA and the initial model was designed pertinent to all the factors and their relevant items (Figure-1). Items with negative loadings and poor loading in each factor were further removed and modification indices were computed after re-running CFA in order to get values closer to an acceptable model fit (Figure-2).

The reliability index (Cronbach [alpha]) of domains 1,2 and 3 established after elimination of items with poor loadings in relevant domains and re-running CFA was computed to be 0.94, 0.88 and 0.71 respectively. However, reliability of modified tool was also found to be acceptable (Cronbach [alpha]= 0.77). Fit indices were computed for both models and their comparison with benchmarks was done (Table-6).

Discussion

Establishing construct validity is actually evaluation of the degree to which statistical and theoretical confirmation support the aptness of inferences.11 Moreover, researchers should try to establish construct validity of an already developing instrument instead of developing and validating a new tool.12 Both EFA and CFA were carried out in the present study for the establishment of construct validity of AMEET inventory by getting this tool filled by the faculty of RMC and its allied hospitals. A similar study was conducted by doctors of a Saudi university medical school that was aimed at establishing the validity of PHEEM inventory by doing EFA.13 Like the present study, the Saudi doctors applied principal component analysis with varimax rotation, and also retained those items in the inventory that had Eigen value greater than 1.13 but they did not do CFA. Psychometric analysis of Jefferson scale of physician empathy14 was done by doing both EFA and CFA.

EFA was done by performing principal component analysis to appraise the relationship between variables and factors. Like the present study, factor co-efficient of 0.40 or greater was selected for retention of variables in this international research.14 But contrary to the current study, factors with Eigen value greater than 1.25 were retained and apart from using AMOS software, structural equation modelling was also done for CFA. Jefferson scale revealed excellent goodness of fit compared to our study. This difference might be due to huge sample of 853 respondents who filled the questionnaire compared to the current sample of 250 doctors. In 2002 a study opined that we should not rely on favourable approximate indices calculated from CFA because the appropriateness of goodness of fit indices might be due to huge variance and low correlation between the variables.15 Moreover, it has been suggested that paradoxical effect of reliability seems to greatly affect confirmatory fit index (CFI).16

A study among 656 Malaysian medical students for the establishment of construct validity of DREEM was done using CFA. This 50-item tool ultimately had 17 items after CFA.17 The current study eliminated 26 items from the AMEET inventory to make it valid for Pakistani culture, while 33 items were removed from the DREEM questionnaire following CFA to make it fit the Malaysian culture and context.17 Contrary to the current study, the Malaysian study was done on a huge sample size of 656. This could be one of the reasons of our inability to establish the construct validity of AMEET inventory. Similar to our study, the faculty of the School of Dentistry in Indonesia worked for the establishment of construct validity of DREEM by getting it filled by 352 medical students. Apart from CFA, Pearson Product Moment Correlation test was applied to test the validity of this tool.18 Ultimately 17 items out of total 50 were sorted out as bad due to weak correlation (r<0.3).

Again, this tool was found to have good reliability (Cronbach [alpha] = 0.883) but its construct validity could not be proven in Indonesian culture and context. The limitation of that study18 was found to be insufficient sample size. Although study participants in the Indonesian research were more than those enrolled in the current study, it failed to determine the construct validity of the tool. This is a matter of great concern and should be scrutinised by further research. Psychometric assessment of DREEM inventory has also been carried out by researchers of Ireland by getting responses of 239 final year medical students.19 Like the present study, it revealed acceptable reliability (Cronbach [alpha] = 0.89) of the inventory, but CFA showed a weak model fit.

Contrary to our study on AMEET inventory within which all domains in the final model showed good reliability, the reliability of one subscale (students' social self-perception) of DREEM had poor reliability (Cronbach [alpha] = 0.55) which also arouses suspicion regarding vagueness of construct validity of DREEM inventory.19 The reason might be the inadequacy of the sample size. Ultimately it was suggested that factor analysis of DREEM inventory should be followed by huge sample full Structural Equation Modelling (SEM) analysis for psychometric evaluation of this instrument.19 IN terms of limitations, the current study had doctors alone as respondents and were likely to have studied in English-medium schools. Therefore, the questionnaire was not translated in Urdu. Besides, data was not gathered by probability sampling technique in order to have a large sample size. In future studies, methods other than factor analysis should also be employed to determine the construct validity of a tool.

In addition, construct validity of this tool should be established in other cultural contexts by getting responses from still more medical teachers so as to get a better model fit. The factor analysis carried out on still large sample size followed by SEM technique is most likely to prove the construct validity of a tool. Moreover, criteria of 5 participants per item need to be revised for authenticity of factor analysis.

Conclusion

AMEET inventory was found to have acceptable reliability. Two indices of CFA met standards, but 4 indices could not, so construct validity could not be established.

Disclaimer: This manuscript is part of the MHPE (Masters in Health Professions Education) research project of one of the authors.

Conflict of Interest: None.

Source of Funding: None.

References

1. Reem A, Ramnarayanan K, Vinod P. Students' Perceptions of Learning Environment in Indian Medical School. BMC Med Educ 2008; 8: 20.

2. Rajan M, Chacko T. Improving educational environment in medical colleges through transactional analysis practice of teachers. F1000Res 2012; 1: 24.

3. Soemantri D, Herrera C, Riquelme A. Measuring the educational environment in health professions studies. Medical Teacher 2010; 32: 947-952.

4. Shehnaz SI, Premadasa G, Arifulla M, Sreedharan J, Gomathi KG. Development and validation of the AMEET inventory: An instrument measuring medical faculty members' perceptions of their educational environment. Medical Teacher 2015; 37: 660-669.

5. Gorsuch RL. Factor Analysis. 2nd ed. Hillsdale, NJ: Erlbaum. 1983.

6. Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. 9th ed. Philadelphia, USA: Wolters Klower Health Lippincott Williams and Wilkins.2012.

7. Messick S. Standards of validity and the validity of standards in performance assessment. Educational Measurement 1995; 14: 5-8.

8. Cronbach LJ, Meehl PE. Construct Validity in Psychological Tests. Psychological Bulletin 1955; 52: 281-302.

9. Ugulu I. Confirmatory factor analysis for testing validity and reliability of traditional knowledge scale to measure university students' attitudes. Educ Res Rev 2013; 8: 1399-1408.

10. MacCallum RC, Widaman KF, Zhang S, Hong S. Sample size in factor analysis. Psychological Methods 1999; 4: 84-89.

11. Messick S. Test validity: A matter of consequences. Social Indicators Research 1998; 45: 35-44.

12. Kimberlin CL, Winterstein AG. Validity and reliability of measurement instruments used in research. Am J Health-Syst Pharm 2008; 65: 2276-2284.

13. Al-Sheikh MH, Ismail MH, Al-Khater SA. Validation of the postgraduate hospital educational environment measure at a Saudi university medical school. Saudi Med J 2014; 35: 734-773.

14. Tavakol S, Dennick R, Tavakol M. Psychometric properties and confirmatory factor analysis of the Jefferson scale of physician empathy. BMC Medical Education 2011; 11: 54.

15. Browne MW, MacCallum RC, Kim, C-T, Anderson BL, Glaser R. When fit indices and residuals are incompatible. Psychological Methods 2002; 7: 403-421.

16. Miles J, Shevlin M. A time and a place for incremental fit indices. Personality and Individual differences 2007; 42: 869-874.

17. Yusoff MSB. The Dundee Ready Educational Environment Measure: A confirmatory factor analysis in a sample of Malaysian medical students. Int J Humanities and Social Sci 2012; 2: 313-321.

18. Leman MA. Construct Validity Assessment of Dundee Ready Educational Environment Measurement (DREEM) in a school of Dentistry. The Indonesian J Med Edu 2017; 6: 11-19.

19. Hammond SM, O'Rourke M, Kelly M, Bennett D, O'Flynn S. A Psychometric Appraisal of the DREEM. BMC Med Educ 2012; 12: 2-5.

Annexure-1: Assessment of Medical Educational Environment by the Teachers (AMEET) Inventory.

Sr.####Statements###SA###A###Neutral###D###SD

###Teachers' perception of teaching

1###I find my role as a teacher interesting

2###There is too much of emphasis on learning factual information

3###The comments given by the teachers help students to improve themselves

4###The preparation that the teachers undertake for doing their classes is adequate

5###The emphasis in classes is on what the teacher does rather than on what the student learns

6###The content and the teaching methods match the learning objectives of the course

7###The teacher respects the students' viewpoints

###Teachers' perceptions of learning activities

8###The learning is "student centered"

9###Students have sufficient opportunities to develop competence

10###Students have sufficient opportunities to develop confidence

11###The students have a clear idea of what they are expected to gain from the course

12###The class environment makes the students want to get actively involved

13###The students are encouraged to be life-long learners

14###Students are encouraged to understand and apply what they learn rather than remember isolated facts

###Teachers' perceptions of students

15###The students are well mannered and respectful

16###The students complete their assignments on time

17###The students are motivated to learn

18###The students come well prepared for their learning activities

19###The students are not sufficiently prepared for the formative assessments (the assessments conducted for giving feedback/practice)

20###The students feel comfortable in learning with other students coming from varied background(s)

###Teachers' perceptions of learning atmosphere

21###The atmosphere in the hospital setting is conducive to learning

22###The time allotted to different learning sessions is appropriate

23###The atmosphere during most of the teaching/learning activities is conducive to learning

24###The students have sufficient opportunities to improve their communication skills

25###The students find the learning experience disappointing

26###The teaching learning experience is very stressful for the students

27###The students find studying medicine enjoyable

28###The students feel comfortable about asking any questions they want

29###The students' feedback about the curriculum is taken into consideration

30###Teachers are encouraged to provide suggestions regarding curriculum

###Teachers' perceptions of collaborative atmosphere

31###Teachers are involved in collaborative efforts related to educational activities

32###Teachers are encouraged to take a trans-disciplinary approach to health problems

33###Teachers have good interpersonal communication skills for dealing with peers and students

34###Teachers are flexible and open to change/suggestions

35###The attitudes of the teachers are conducive to effective team work

36###Teachers dominate over subordinates, peers or students

37###I am satisfied with the opportunities that I get for working with other faculty

38###There is a formal support system for faculty who get stressed

39###Teachers have sufficient time to plan their teaching activities

###Teachers' professional self-perceptions

40###I am knowledgeable in educational concepts for my role as a teacher

41###I possess the necessary teaching skills for undertaking my duties

42###I have effective communication skills for dealing with hospital patients

43###Opportunities are available for trying out alternative approaches in teaching

44###I have opportunities for learning research methodology and research communication skills

45###I have opportunities for developing the skills of educational administration

46###The Faculty Development Programs are adequate in preparing me for the tasks expected from me

47###The teachers adopt a variety of teaching methods to cater to diverse styles of learning

48###Adequate teaching and learning resources (including physical and material) are available

49###The students have sufficient opportunities to discuss their learning problems with the faculty

50###I am encouraged to go to conferences that improve my knowledge and skills

Annexure-2: Items Eliminated from AMEET Inventory.

Item No.###Statements

Item 1###I find my role as a teacher interesting

Item 2###There is too much of emphasis on learning factual information

Item 3###The comments given by the teachers help students to improve themselves

Item 5###The emphasis in classes is on what the teacher does rather than on what the student learns

Item 8###The learning is student centered

Item 12###The class environment makes the students get actively involved

Item 13###The students are encouraged to be lifelong learners

Item 14###The students are encouraged to understand and apply what they learn rather than remember isolated facts

Item 16###The students complete their assignments in time

Item 17###The students are motivated to learn

Item 19###The students are not sufficiently prepared for the formative assessments

Item 24###The students have sufficient opportunities to improve their communication skills

Item 25###Students find learning experience disappointing

Item 26###The teaching learning experience is very stressful for the students

Item 29###The students' feedback about the curriculum is taken into consideration

Item 31###Teachers are involved in collaborative efforts related to educational activities

Item 32###Teachers are encouraged to take trans-disciplinary approach to health problems

Item 34###Teachers are flexible and open to change and suggestions

Item 36###Teachers dominate over subordinates. peers or students

Item 37###I am satisfied with opportunities that I get for working with other faculty

Item 38###There is a formal support system for faculty who get stressed

Item 43###Opportunities are available for trying out alternative approaches in teaching

Item 44###I have opportunities for learning research methodology and research communication skills

Item 45###I have opportunities for developing skills of educational administration

Item 46###The faculty development program are adequate in preparing me for the tasks expected from me

Item 47###The teachers adopt a variety of teaching methods to cater to diverse styles of learning
COPYRIGHT 2019 Knowledge Bylanes
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2019 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Shahid, Rizwana; Khan, Rehan Ahmed; Yasmeen, Rahila
Publication:Journal of Pakistan Medical Association
Article Type:Report
Date:Jan 31, 2019
Words:4578
Previous Article:Establishing age specific spirometry reference ranges for children/adolescents of Karachi, Pakistan: Randomized trials.
Next Article:Feasibility of robotic surgery in a developing country, a public sector perspective.
Topics:

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters