A modification of the Schutte Emotional Intelligence Scale using multivariate generalizability theory.
The increasing amount of cross-cultural psychological research being conducted means there is a growing need for standardized and validated practices for translating instruments (Van de Vijver & Hambleton, 1996; Van de Vijver & Leung, 1997). The aim of cross-cultural adaptation is to ensure consistent content and face validity between the source and target versions of a questionnaire or survey (Beaton, Bombardier, Guillemin, & Ferraz, 2000).
Wang and He (2002) translated the EIS into Chinese and used classical test theory (CTT) to assess the scale's internal consistency reliability, obtaining a Cronbach's alpha value of .84. Y. H. Huang, Lu, Wang, and Shi (2008) then examined the validity of the Chinese version of the EIS and determined that the four-dimensional scale structure had the best model fit. However, CTT has limitations, such as inadequate reliability estimation and inflated measurement error. Thus, determining scale reliability using the CTT does not guarantee the validity of a measure.
Compared to the CTT, generalizability theory (GT; Cronbach, Rajaratnam, & Gleser, 1963) contains principles and methods of experimental design, and features more statistical adjustment techniques. For instance, GT involves calculation of within-person variance and its interaction with both measurement error (i.e., phrasing of an item) and occasion variance (e.g., a negative event that massively impacted on emotion regulation; Shavelson & Webb, 2005). GT analysis comprises two steps: a generalizability study (G study) and a decision study (D study). The main goals of a G study are to analyze and estimate the association between various factors, establish the number of factors, and the initial measurement design process. The main goals of a D study are to explore changes in the generalized coefficient and reliability indices when the number of items in the measure changes. Brennan (1992) developed GT into univariate generalizability theory and, from this, proposed multivariate generalizability theory (MGT), in which analysis of measurement errors is performed in greater depth, so that more accurate results are obtained in terms of scale reliability. MGT features pluralistic concepts of reliability and reliability estimation, so it is more suitable for analytically processing the multidimensional reliability of a test. MGT has the unique function of identifying scales for which the overall reliability is high but the reliability of certain dimensions is low, then providing suggestions for improvement (Xu, 2009; Yang & Zhang, 2003).
Thus, Zheng, Gu, and Zhu (2009) adopted MGT to analyze the EIS and found that the generalization coefficient (GC) and dependent index (DI) of the scale's composite universe score were both high (GC = .846, DI = .827). Zheng et al. further reported that the AOE, ROE, and ROOE dimensions had good reliability, whereas UOE had poor reliability. Our aims in the current study were to extend these previous MGT-based studies by identifying existing problems with the EIS (e.g., inaccurate factor-weight ratio determination) and developing a modified version with improved reliability and validity.
The G study participants consisted of 230 students randomly selected from three schools in Shanghai and Anhui Province, China. Of the 223 retrieved survey forms, 210 (n = 95 men and 115 women, aged from 14 to 20 years; M = 17.10, SD = 3.46) were valid.
The participants for item, factor, reliability, and validity analyses comprised 1,450 students randomly selected from five educational institutions, comprising one middle school, two high schools, and two universities in Shanghai and Anhui Province, China. Of 1,378 survey forms we recovered, 1,339 were deemed valid and included in the analysis phase. The participants' gender was approximately evenly distributed between men and boys (n = 629) and women and girls (n = 710), and ages ranged from 12 to 23 years (M = 17.26, SD = 4.79). Participants were surveyed in batches and we performed exploratory factor analysis (EFA) on the 835 forms collected in the first batch and confirmatory factor analysis (CFA) was performed on the second batch of 504.
All participants provided written informed consent and parental consent was obtained for students under the age of 16 years. The study protocol was approved by the Ethics Committee of Shanghai Normal University.
We revised the EIS in the following five stages: initial translation into Chinese, synthesis of the translation, back-translation into English, expert committee review, and testing of the prefinal version. Per existing guidelines for the process of cross-cultural adaptation of self-report measures (Beaton et al., 2000; Cortina, 1993), we deleted some of the items, added other items, and modified the content of items that we judged to be not suitable for the Chinese context. The draft modified EIS we used comprised 52 items.
We used CTT to perform item analysis of the first draft, with item scores and total correlations applied to conduct item discrimination. On the basis of the item discrimination and EFA results, we removed 13 items with discrimination values lower than .30, as well as one item that was not grouped into any specific dimension. Thus, a revised test with 38 items across four dimensions was formed. We applied MGT to test the second draft, with a G study and a D study being used to acquire information concerning the maximum number of items for each subtest. Next, separate groups of participants were recruited for the item analysis and EFA, at which point six items were deleted and the remaining items were used to constitute the formal test, comprising 32 items across four dimensions. Finally, we assessed the reliability and validity of this formal test.
We adopted the multiple participant x test item (p x i) random measurement model to determine the emotional intelligence level of participants; specifically, the items in the test are measured to determine whether they interact. The specific approach entails setting the test scores as the dependent variable, p and i as random variables, and p x i as the interaction term, then applying the general factorial method in generalized linear modeling using SPSS. In our study, we used SPSS 19.0 and we used LISREL 8.80 statistical software to process the data.
In multiple p x i random measurements, the participants can be regarded as being randomly selected from an infinite pool, and items can also be seen as a sample randomly selected from an infinite number of items. Data collected using this approach can be assessed with a two-factor random-effects analysis of variance.
The G study results indicated that among the variance components of the four dimensions, AOE had the largest values, followed by UOE, ROOE, and then ROE (see Table 1). However, the difference between the four variance components was not significant because the largest minimum difference did not exceed .074. Thus, from an individual perspective the variance components of the four dimensions were still relatively balanced, although their size was slightly different.
When combined with the interfactor covariance and correlation coefficients, we found the covariance between AOE and UOE to be relatively small (.17), whereas the covariance of the remaining factors was larger (.24-.26). This indicates that a certain level of independence exists between the AOE and UOE factors, whereas the other factors have a higher degree of intercorrelation. In addition, the degree of variance in emotional intelligence that was explained by the tested items was smaller than that of the subjects, which indicates that the measure has high reliability and accuracy with very small error levels.
We considered the estimated variance and covariance matrix in the G study and, therefore, conducted a further estimation analysis regarding the universal scores and corresponding variance components in the four dimensions, and then the GC, DI, and, as per Yang and Zhang (2003), the ratio of the sum of the variance of the measurement target and the relative error of the measurement (relative signal-to-noise ratio), and the ratio of the sum of the measurement target variance and the sum of the measurement absolute error variance (absolute signal-to-noise ratio).
Table 2 shows that both the GC and the DI of the four dimensions of the EIS reached an ideal level, confirming that the EIS comprises four dimensions and is feasible for measuring emotional intelligence in the Chinese cultural context, and that the four-dimensional scale is the best fit compared to scales with three dimensions or fewer. In addition, the GC and DI of the composite universe score were significantly higher than those of the four dimensions, and the error level was relatively small, which indicates that the EIS has good overall reliability.
In the G study, the low covariance between the AOE and UOE factors means that whether the four dimensions can be combined for use as a total scale cannot be determined. However, the results of the D study indicate that synthesizing the four dimensions into a total scale significantly increased the measurement reliability, suggesting that this approach is feasible. From the results set out in Table 2 it can also be seen that the UOE dimension of the revised EIS had a higher GC and DI than those reported for the original EIS, indicating that the measurement accuracy of the revised EIS is superior to that of the original scale. In the original EIS there are only four items to assess UOE, resulting in a low GC (Zheng et al., 2009). To determine the optimal number of items, we conducted the D study based on the number of items in each dimension (see Table 3). The results indicated that the GC of the UOE dimension was acceptable when the number of items ranged from six to eight, which was further supported by the GC of the total scale (see Table 3). The results for the other three dimensions showed that the number of items for each dimension should be maintained at six to eight to achieve a high GC and to lower costs for administering the scale.
The findings of the D study also indicate that the GC was acceptable when the modified EIS comprised four dimensions with six subtest items in each dimension. There should be more than seven items per dimension if the GC of each dimension exceeds .700. However, when the number of items per dimension was increased to eight the total GC increased very little (< .005). Therefore, the GT analysis results indicate that, after performing pruning based on CTT theory, the 38 test items should be reduced further to make the scale more concise.
Item and Factor Analysis
In the next stage of analysis, the selection rate of each item was checked, then we checked the difficulty and distinction levels of the items. After applying the above criteria, we removed three items with distinction levels under .35.
We performed an EFA with varimax rotation to maximize the variance of the squared loadings of each factor in the matrix, which allowed us to identify the underlying structure of the original variables. The EFA of the second draft of the EIS, which was conducted with a large sample of 835 participants, revealed that the measure should consist of four factors, and three items were removed because of low factor loadings. Consequently, the formal test comprised 32 items.
We then assessed the suitability of the data for factor analysis and obtained the following results: approximate chi square of Bartlett's test of sphericity = 5897.48, degrees of freedom (df = 496, Kaiser--Meyer--Olkin (KMO) measure of sampling adequacy = .91. The significance (p < .001) of Bartlett's test of sphericity and KMO values above .60 were acceptable. After performing principal components factor analyses, we found that the rubble figure stabilized the main starting point from the third factor according to the gravel diagram and the percentage of variance explained. We also used parallel analysis to further determine that the most suitable number of factors was four; compared to gravel testing, in parallel analysis the influence of researchers' subjectivity is reduced, so that the results are more objective, accurate, and valid (Horn, 1965). Finally, we extracted four factors covering the 32 items, which explained 48.12% of the total variance in emotional intelligence (AOE, ROE, ROOE and UOE factors explained 26.01%, 9.35%, 7.47%, and 5.29%, respectively, of the variance). Therefore, we included these four dimensions in the revised EIS. The factor loadings of each item ranged between .40 and .74.
Reliability and Validity Analysis
We calculated Cronbach's alphas estimates for the EIS dimensions with a sample of 1,339 participants and found that these ranged from .70 to .78 and the Cronbach's [alpha] for the overall scale was .88.
We also tested 75 participants and retested them 3 weeks later. The lowest test-retest reliability was found for the dimension of ROOE and the AOE dimension had the highest test-retest reliability. The test-retest reliability coefficients and the homogeneity reliability results are shown in Table 4.
We evaluated the structure of the scale via CFA of the data from the sample of 504 participants. We used structural equation modeling to test the construct validity of the revised EIS, employing a maximum likelihood estimation procedure that is robust to violations of normality. The fit indices were as follows: chi square ([chi square]) = 952, df = 460, [chi square]/df = 2.07, root mean square error of approximation = .046, normed fit index = .91, nonnormed fit index = .95, comparative fit index = .95, goodness-of-fit index = .90, incremental fit index = .95. The results indicate that the four dimensions of the scale provided a good fit for the data. Thus, the revised EIS has good structural validity.
We aimed to modify and improve the widely used EIS in a Chinese cultural context by applying MGT, thereby providing a new perspective for the study of psychometric assessment tools. In multivariate generalization studies, the main goals of the G study are to identify as many sources as possible of potential error in emotional intelligence measures and to estimate the size of the variance components of these sources of error (Cronbach et al., 1963). In contrast, the main goals of the D study are to deduce or explain the measurement results according to the specific decision needs, reconstruct a variety of generalized regions, estimate the size of the variance components at the level of the sample mean, and then estimate the various measurement errors and measurement accuracy indices (relative error and GC or absolute error and DI) to provide valuable information that can be used to improve measures (Yang & Zhang, 2003). Our results for the modified EIS showed significant decreases in both relative and absolute error variance when compared with the Chinese version developed by Zheng et al. (2009). The D study can be used to evaluate the quality of postevaluations and to explore the quality of pretests for the implementation of large-scale formal tests and for providing improvement suggestions. Thus, MGT provides a technique for accurately estimating the reliability of psychometric measures.
The results of our study provide further evidence regarding the ideal structural dimension of the EIS and the optimal number of items per dimension. By investigating the dimensionality of the modified EIS, we have confirmed its theoretical structure. The integrated CTT and MGT analysis method adopted in this study constitutes a useful approach for evaluating the dimensionality of the EIS, whereby the impacts of dimensions can be empirically evaluated by splitting the data into subsamples for EFA and CFA. We caution, however, that large numbers of participants are required when using these methods, so that the models can be adequately fitted to each random sample. Thus, in many cases these methods may not be feasible.
In contrast, generalized theoretical analysis requires only a small number of participants, and data can be analog processed to establish the most suitable number of dimensions and number of items for each dimension. To study methods of improving the EIS we opted to change the sample size of each dimension, which allowed us to observe the changing characteristics of the measure's reliability. As can be seen in Table 3, reducing the number of items in the UOE dimension to one reduces the total GC to .895. Although this reliability result is acceptable, it is generally impossible for a dimension to have just one item. When the capacity of the items increase, the GC also increases, which shows that increasing the number of items in the EIS can improve the scale's reliability; this is consistent with the CTT research conclusion (Ciarrochi et al., 2001).
Limitations and Future Research Directions
Like many psychometric measures, MGT has its own limitations. Because it is based on a random sampling model, owing to the variability of sampling, the analyzed results are only a description of the statistical law of the test, and the MGT's reliability depends heavily on the completeness of the measured data. Therefore, researchers must have a high level of control over the test design and actual measurement. Moreover, Sharma, Deller, Biswal, and Mandal (2009) found significant differences in the factor structure of the EIS across different cultures We recommend that future researchers select cross-cultural samples to further validate our results.
It is also necessary to apply diversified measurement patterns and advanced statistical methods of scale design and verification. We did consider diversity of participants and consistency of the application of different methods; however, we did not perform item response theory measurements. In future studies an item response curve might be used for item response analysis, which may guide item screening and determine comparing fractions (M. M. Huang & Wang, 2015); further, if conditions permit, differential item functioning could be applied to judge possible deviations in EIS test items and ensure the fairness of the test. In addition, a practical study of the EIS test should be performed, incorporating a combination of cross-sectional and longitudinal study methods.
From this study of the application of MGT to the four dimensions of emotional intelligence, the following conclusions can be drawn. Compared with the original version, our modified EIS has better reliability and validity in the Chinese cultural context, and each dimension has a more balanced proportion of universe variance components. The MGT provides a realistic view of multiple reliability and multivariate reliability estimation methods, meaning it can be applied to the different purposes of the measures, allowing for improved estimation of the same test's multiple measurement reliability. This is of great importance for revising measurement scales.
This study was supported by Shanghai Education Scientific Research (C17064), the Humanities and Social Sciences Research Project, the Key Projects of Philosophy and Social Sciences Research (13JZD046), the National Natural Science Foundation of China (31760286 and 31470997), and the Education and Teaching Reform and Construction Project "Social Psychology".
Beaton, D. E., Bombardier, C., Guillemin, F., & Ferraz, M. B. (2000). Guidelines for the process of cross-cultural adaptation of self-report measures. Spine, 25, 3186-3191.
Brennan, R. L. (1992). Generalizability theory. Educational Measurement Issues and Practice, 11, 27-34. https://doi.org/bd6rm9
Ciarrochi, J., Chan, A. Y.-C., & Bajgar, J. (2001). Measuring emotional intelligence in adolescents. Personality and Individual Differences, 31, 1105-1119. https://doi.org/fc59hp
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and application. Journal of Applied Psychology, 78, 98-104. https://doi.org/fn7g4t
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Mathematical and Statistical Psychology, 16, 137-163. https://doi.org/dk3m7b
Dumitrescu, A. L., Badita, D., Dogaru, C. B., Toma, C., Pertea, G., & Duta, C. (2014). Psychometrical properties of the Romanian version of the Schutte Emotional Intelligence Scale. Procedia--Social and Behavioral Sciences, 159, 557-560. https://doi.org/cp8h
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185. https://doi.org/b7684f
Huang, M. M., & Wang, L. J. (2015). Item response theory analysis based on the Schutte Emotional Intelligence Scale [In Chinese]. Psychology: Techniques and Applications, 3, 21-25. https://doi.org/cqpv
Huang, Y. H., Lu, A. Q., Wang, L., & Shi, J. Q. (2008). Validation of the Schutte Emotional Intelligence Scale [In Chinese]. Acta Scientiarum Naturalium Universitatis Pekinensis, 44, 970-976. https://doi.org/cqpt
Lu, J. M. (2005). Discussion of basic conception of emotional intelligence [In Chinese]. Psychological Science, 5, 1246-1249. https://doi.org/cqpw
Schutte, N. S., Malouff, J. M., Hall, L. E., Haggerty, D. J., Cooper, J. T., Golden, C. J., & Dornheim, L. (1998). Development and validation of a measure of emotional intelligence. Personality and Individual Differences, 25, 167-177. https://doi.org/fmng7j
Sharma, S., Deller, J., Biswal, R., & Mandal, M. K. (2009). Emotional intelligence: Factorial structure and construct validity across cultures. International Journal of Cross Cultural Management, 9, 217-236.
Shavelson, R. J., & Webb, N. M. (2005). Generalizability theory. Encyclopedia of Social Measurement, 2, 99-105. https://doi.org/cfnfjx
Van de Vijver, F., & Hambleton, R. K. (1996). Translating tests. European Psychologist, 1, 89-99. https://doi.org/c5xd2z
Van de Vijver, F., & Leung, K. (1997). Methods and data analysis for cross-cultural research. Thousand Oaks, CA: Sage.
Wang, C. K., & He, Z. W. (2002). The relationship between parental rearing styles and middle-school students' self-efficacy and emotional intelligence [In Chinese]. Chinese Mental Health Journal, 16, 781-782. https://doi.org/cqps
Xu, L. (2009). The development and application of a revised version of the Schutte Emotional Intelligence Scale [In Chinese] (Unpublished master thesis). Shanghai Normal University, China.
Yang, Z. M., & Zhang, L. (2003). Generalizability theory and its applications [In Chinese]. Beijing, China: Educational Science Publishing House.
Zampetakis, L. A., Kafetsios, K., Lerakis, M., & Moustakis, V. S. (2017). An emotional experience of entrepreneurship: Self-construal, emotion regulation, and expressions to anticipatory emotions. Journal of Career Development. 44, 144-158. https://doi.org/cqpx
Zheng, X., Gu, H., & Zhu, P. (2009). A study of the Schutte Emotional Intelligence Scale using multivariate generalizability theory [In Chinese]. Psychological Science, 32, 181-183. https://doi.org/cqpr
Lei Xu (1), Zhan Liu (2), Xianliang Zheng (3), Hai-Gen Gu (4), Jiamei Lu (4), Wenhai Zhang (5)
(1) Electrical College, Shanghai Dianji University and College of Education, Shanghai Normal University
(2) Mechanical College, Shanghai Dianji University
(3) College of Education, Gannan Normal University
(4) College of Education, Shanghai Normal University
(5) Mental Health Center, Yancheng Institute of Technology
How to cite: Xu, L., Liu, Z., Zheng, X., Gu, H., Lu, J., & Zhang, W. (2019). A modification of the Schutte Emotional Intelligence Scale using multivariate generalizability theory. Social Behavior and Personality: An international journal, 47(1), e7419
CORRESPONDENCE Jiamei Lu, Education College, Shanghai Normal University, 100 Guilin Road, Shanghai 200234, People's Republic of China. Email: firstname.lastname@example.org or to Wenhai Zhang, Mental Health Center, Yancheng Institute of Technology, Yancheng 224051, People's Republic of China. Email: email@example.com
Table 1. Estimation of Variance and Covariance Components in the Generalizability Study of the Modified Schutte Emotional Intelligence Scale Effect AOE ROE ROOE UOE Participant (p) .26 .22 .28 .18 .24 .19 .29 .23 .25 .29 .21 .21 .17 .24 .19 .22 Test item (i) .02 .19 .08 .05 p x i .68 .90 .55 .65 Note. Variance components in the corresponding factor estimation of the effect appear on the diagonal, interfactor covariance component estimates appear below the main diagonal, and interfactor correlation coefficient estimates appear above the main diagonal. AOE = awareness of emotion, ROE = regulation of own emotions, ROOE = regulation of others' emotions, UOE = use of emotion. Table 2. Decision Study Results for the Modified Schutte Emotional Intelligence Scale Based on Multivariate Generalizability Theory Index AOE ROE ROOE UOE CUS Universe variance component .26 .19 .21 .22 .23 Relative error variance .07 .09 .06 .07 .02 Absolute error variance .07 .11 .06 .07 .02 Mean error variance .01 .02 .01 .01 .01 Generalizability coefficient .80 .68 .79 .78 .94 Dependent index .79 .64 .77 .76 .92 Relative signal-to-noise ratio 3.91 2.10 3.85 3.44 15.07 Absolute signal-to-noise ratio 3.78 1.75 3.38 3.16 11.55 Note. AOE = awareness of emotion, ROE = regulation of own emotions, ROOE = regulation of others' emotions, UOE = use of emotion, CUS = composite universe score. Table 3. Problem Capacity and Generalizability Coefficient Variation for the Use of Emotion Dimension of the Modified Schutte Emotional Intelligence Scale Index Use of emotion Topic capacity 1 2 4 6 8 10 12 14 Generalizability .26 .41 .58 .67 .73 .78 .81 .83 coefficient Total generalizability .90 .90 .90 .91 .91 .92 .92 .92 coefficient Table 4. Test-Retest Reliability and Homogeneity of the Modified Schutte Emotional Intelligence Scale Index AOE ROE ROOE UOE EIS Number of items 8 8 8 8 32 Test-retest reliability .74 (*) .70 (*) .69 (*) .71 (*) .84 (*) Internal consistency .78 .71 .70 .72 .88 reliability (Cronbach's [alpha]) Note. AOE = awareness of emotion, ROE = regulation of own emotions, ROOE = regulation of others' emotions, UOE = use of emotion, EIS = Schutte Emotional Intelligence Scale. (*) p < .01.
|Printer friendly Cite/link Email Feedback|
|Author:||Xu, Lei; Liu, Zhan; Zheng, Xianliang; Gu, Hai-Gen; Lu, Jiamei; Zhang, Wenhai|
|Publication:||Social Behavior and Personality: An International Journal|
|Date:||Jan 1, 2019|
|Previous Article:||When independents favor far extension: Self-construal, brand extension, and brand concept consistency.|
|Next Article:||Disruptive behaviors in online games: Effects of moral positioning, competitive motivation, and aggression in "League of Legends".|