Culture-based extreme response bias in surveys employing variable response items: an investigation of response tendency among Hispanic-Americans.
Survey response biases can render results from a number of statistical tests completely spurious, especially those from analyses based on survey data employing Likert-type items. This problem is particularly endemic to international studies and multiple studies over the years have suggested that cultural traits in a society lead to response bias. There is some evidence that response bias is a problem in Latin cultures, such as that dominant in Mexico. Thus far there has been very little work examining whether Hispanics in the United States are subject to survey response bias. This study used a sample of 316 university students, roughly half of whom identified themselves has Hispanic, to test whether extreme response tendency and/or midpoint bias could be shown in that American subpopulation. Results of a Multivariate Analysis of Variance procedure showed that compared to non-Hispanics, Hispanic respondents exhibited not only extreme response bias (the tendency to use "1"s and "7"s more often on a seven-point scale) but also used significantly fewer midpoints on the scale.
Empirical work in the field of International Business must often bridge a large cultural chasm that raises a number of methodological difficulties. One of the most formidable is the pervasive use of Likert-type and semantic differential rating scales and their long-recognized susceptibility to culture-related response bias ENRf8(Adler, Campbell, & Laurent, 1989; Jaccard & Wan, 1986; Leung & Bond, 1989; Mullen, 1995; Zax & Takashi, 1967). The implications of ignoring response tendency differences can be profound, leading to major inferential errors in cross-cultural research ENRf8(Chun, Campbell, & Yoo, 1974; Cronbach, 1946; Singh, 1995).
For example, Adler and colleagues ENRf8(1989) encountered such unusual response distributions for Likert-type items among P.R.C. managers in their study that they abandoned substantive inquiry addressing U.S.-P.R.C. differences in managerial attitudes altogether and analyzed instead methodological barriers to cross-cultural research. Unfortunately, the literature addressing cultural differences in response tendencies is relatively undeveloped, in part because cross-cultural studies often ignore this problem, and indeed measurement concerns altogether ENRf8(Singh, 1995). This is despite major inferential limitations and distortions that are operant, whether acknowledged or not. If present, cultural response bias distorts statistical analysis by : (1) rendering group mean differences uninterpretable, (2) spuriously raising or lowering indexes of a measure's internal consistency, (3) spuriously affecting correlations between variables and related techniques such as regression, and (4) affecting the results of methods assessing underlying dimensions, such as factor analysis ENRf8(Chun et al., 1974).
Many subfields within the field of International Business make heavy use of survey methodology. Surveys are typically distributed to respondents in more than one country, very often including the US, for comparison purposes. These surveys are commonly used to determine how those from other cultures differ from Americans on one or more behavioral dimensions. The surveys may be given in English to English-speaking foreigners or translated into native languages. Regardless of which form the survey may take, response data from more than one country are often not directly comparable due to the problem of response bias.
Response bias refers to the fact that survey-takers have been observed to respond to typical Likert-type or semantic differential scale items (items that employ multiple-response scales; e.g., "I am satisfied with my job"--respondent chooses "1" = strongly disagree, "2" = moderately disagree, etc.) in very different ways based on their cultural background. For example, Americans are thought to exhibit a "midpoint bias" when confronted with a statement with which one must indicate some measure of agreement or disagreement on five-point or seven-point scale, they will tend to circle middling responses around the neutral point on the scale. By contrast, respondents from cultures in Asia Pacific countries such as China are thought by some to exhibit an extreme response style, picking numbers toward either extreme of the scale. This affects the measurement of the construct a researcher is trying to assess, such as individual levels of job satisfaction or organizational commitment.
This measurement distortion--if unrecognized- leads to a number of other methodological problems such as (1) failure to capture variable relationships assessed with correlational techniques such as regression, (2) false positive or false negative results in difference tests comparing cultures on some attribute, and (3) the distortion of factor-analytic results used to examine the reliability and validity of scales that are used. In other words, cross-cultural work that takes no cognizance of potential response bias runs the risk of obtaining completely spurious findings. For example, we have encountered studies comparing survey responses from Americans on a large number of variables with responses from one of the East Asian countries which found the other country's means to be higher on every single variable. Findings were reported as completely substantive with no mention of the possibility of extreme response tendency on the part of the other country's respondents. This is despite the fact that the East Asian countries are thought prone to this form of response bias.
Although investigations of cultural response style have been conducted in a variety of countries, deeper focus on particular countries has been lacking. Given the large volume of trade with Mexico, greatly increased since the implementation of the NAFTA agreement, knowledge about the effects of response bias on survey methodology is particularly deficient. In addition, the number of people within the United States comprising the Mexican-American subculture is increasing rapidly. What little work that has been done so far has indicated that response bias, specifically extreme response tendency, may be a problem in Latin-American samples ENRf8(Stening & Everett, 1984).
Information about whether Hispanics are also characterized by some form of survey response bias would be valuable first because it, by extension, gives us some idea of the nature of response bias in Latin cultures outside the U.S. This evidence also has obvious importance for any research involving populations within the U.S. employing survey-based data because a sample that includes a relatively large proportion of Latinos, when compared to a sample with a much smaller proportion, would yield spurious results. This is because there would be significant differences in responses between the two samples based merely on the response style artifact, even if there were in fact no substantive differences between the two samples on the variables being studied.
Some previous work suggests that survey response bias among Hispanic-Americans make take the form of extreme response tendency ENRf8(Hui & Triandis, 1989). If this is in fact the case, then regardless of a survey's substantive focus, Hispanics would tend to choose "1"s and "7"s on a seven-point Likert-type scale significantly more often than do non-Hispanic Americans. Or similarly, they might choose "1"s and "5"s more often when presented with a five-point scale. Unfortunately, the paucity of work in this area is insufficient to provide any confidence that this response bias is indeed characteristic of Hispanics. Additionally, very little is known about other demographic variables that may be a factor, such as age and education.
The Triandis study used a very small sample of 59 Hispanics and 60 non-Hispanics and did find evidence for extreme response tendency among Hispanics, although this was limited to item stems using five-point variable response scales. When the same item stems were employed with ten-point response scales there was no statistical significant difference in the number of scale endpoints chosen by the two groups. Another study also provided some evidence for extreme response bias on the part of Hispanics when compared with non-Hispanics (Martinez & Johnston, 1992) but there was very little information given as to the nature of the scales, item content, and item format. The paucity of work done in this area means that the existence of extreme response style among Hispanics in the United States is still very much an open question.
In addition, there has been no study of whether non-Hispanic Americans may display a midpoint bias when compared with Hispanic Americans. It stands to reason that if Hispanics favor scale endpoints, then midrange responses will be chosen less frequently. Non-Hispanic Americans would then tend to exhibit, relative to Hispanics, a midpoint bias. There is some evidence that Americans are prone to choosing scale midpoints when compared with East Asians, e.g. the Chinese, but this American response tendency has never been explored in relation to Hispanic-American subpopulations.
Finally, what little work has been done suggests that, among demographic variables, age, education levels, and the degree of acculturation are related to the tendency of Hispanics to employ an extreme response style. Specifically, those Hispanics who are older, less educated, or have undergone acculturation to the broader American culture may display a greater extreme response tendency than those who are younger, more educated, and/or more acculturated. Whether Hispanics and non-Hispanics of roughly the same age and education level would differ in their survey response patterns has been scarcely addressed.
With the above questions in mind, we sought to examine in this study whether Hispanic survey respondents exhibit extreme response bias compared to non-Hispanic respondents--reflected in atypically extreme responses for survey items using a multiple-response format (e.g., marking a "1" or "7" on a seven-point scale). We also investigated whether the two groups differed significantly in their preference for the scale midpoint. Finally, whether any extreme response tendency that might be shown varied as a function of demographic variables.
A total of 33 items stem were used in this study (see Appendix). Nine of the items were taken from taken from two scales common in the comparative management area of cross-cultural research, one measuring locus of control and the other, collectivism. Also included were items taken from scales included in a cross-cultural study investigating American, Korean, and Chinese samples (Chen, Lee, & Stevenson, 1995). Semantic anchors for the variable response scale associated with each item stem were "1" completely disagree, "2" moderately disagree, "3" slightly disagree, "4" neither agree or disagree (neutral), "5" slightly agree, "6" moderately agree, and "7" completely agree. All survey items and instructions were translated by a bilingual Mexican studying in the United States. Back-translation ENRf8(Brislin, 1970) into English was carried out by two bilingual Americans. Survey respondents were given a choice of English versus Spanish surveys, but none opted for the Spanish-language survey. This may be construed as an indicator that the Hispanics in our study were fairly well acculturated to the broader American culture, at least with respect to their English skills.
The sample was comprised of 316 college juniors, seniors and MBA students enrolled at two universities in the Southwest. The average age was 26.74. The sample was split almost exactly between males and females (51 and 49% respectively) and between Hispanics and non-Hispanics (50% each). Forty percent were employed full time.
Initially, we tabulated frequency distributions of response choices for each country group by item and these suggested that substantial differences in response patterns existed. Next, an "extreme response" score was calculated for each respondent indicating the number of times a "1" or "7" response was chosen and a "midpoint response" score computed for the number of "4"s selected. The count of extreme scores and midpoint scores for each respondent was cumulative across all items.
Next, response patterns were compared by cultural group to assess whether Hispanic subjects assigned extreme and midpoint scores differently than non-Hispanic subjects. A Multivariate analyses of variance (MANOVA) procedure was conducted to control Type I error while analyzing between-group mean differences in extreme and midpoint response scores. The MANOVA was used because the chance of observing a significant difference between groups for multiple dependent variables using difference tests for each outcome, assuming no true population differences, increases according to the number of tests performed. MANOVA controls Type I error by testing all dependent variables simultaneously for significant differences across independent variable categories. While comparisons between groups were limited to two, use of the MANOVA represented a more conservative test than mere computing difference tests, such as T-tests. The two dependent variables were extreme and midpoint frequency scores for the total pool of scale items. Cultural group served as the independent variable. The Wilks-Lambda test provided an index of whether culture groups differed in relation to both dependent variables combined. Univariate F-tests computed in connection with MANOVA determined whether extreme and midpoint scores differed across culture groups.
MANOVA indicated overall differences (Wilks-Lambda test = p < .01, Table 2) between Hispanic and non-Hispanic respondents for extreme and midpoint response scores analyzed in combination. Consistent with expectations, F-tests (p< .05) performed in connection with the MANOVA procedure indicated that Hispanic respondents selected extreme response categories ("1" or "7") more frequently than did non-Hispanics. Interestingly, non-Hispanics chose the midpoint significantly more frequently (p< .05) than did Hispanics. Intercorrelations among study variables revealed that of the three demographic variables, age, sex, and number of family dependents, only age was significantly correlated with one of the response score variables, and this was a negative relationship with extreme response score. In other words, older subjects were significantly more likely to exhibit extreme response bias.
This study represented an exploratory foray into the poorly understood area of culture-based response bias among Latin cultures in general and Hispanics in the United States in particular. This line of research is still embryonic and even the most basic principles have yet to be established. Two such principles that may well be established eventually are that survey respondents in Latin cultures are prone to extreme response bias and that this extends to Latino subcultures in the United States. The findings of this study provide indirect evidence that the former principle is true, and direct evidence that the latter is true. Specifically, our results indicate that in the United States, those identifying themselves as Hispanic respond to Likert-type survey items in a way that is substantially different than American non-Hispanics.
The implications of such a difference in survey response patterns are potentially far reaching. The number of studies in business-related fields that survey American samples with Likert-type variable-response format items is quite large, not to mention the number that would apply for all academic fields combined. A substantial portion of such studies is likely, given the considerable and ever-increasing number of Hispanics in the United States, to include large numbers of this groups in sample populations under study. These studies run a significant risk of obtaining spurious findings when correlational techniques are used, and this usage use is quite common. Correlational techniques distorted by response bias would include commonly used procedures such as ordinary least squares regression, logistic regression, and structural equation modeling. Scale development studies employing exploratory or confirmatory analysis would bee affected as would be all studies that employ coefficient alpha tests of scale reliability. In addition, studies that compare more than one sample to another run a sizeable risk of obtaining significant differences between the samples, even given no actual substantive differences, based on extreme or midpoint response effects that arises when one sample has substantially higher proportion of Hispanics.
There have been no studies as of yet indicating that Americans from the broader "Anglo" culture exhibit a midpoint response compared to those from the Latino subculture. Neither have there been studies showing that this is the case for other countries in the Anglo group, such as Great Britain, Canada, Australia, and New Zealand. This is undoubtedly due to the fact that research focused on culture-based response bias is still in its early stages. Our study indicated such a pattern and this has many of the same implications for spurious research findings as those for extreme response bias. It should be pointed out that finding an Anglo midpoint response bias vis-a-vis Latino response patterns is not the converse of finding a Latino extreme response bias when compared with Anglo responses.
Age was significantly correlated with extreme response scores for the sample. This is in line with results from studies employing Latin and Asian samples. The reason for the correlation between age and extreme response is still largely unknown although there has been speculation that older members of cultural groups are more culture bound. The thinking is that in societies that are gradually undergoing social and cultural change, it is naturally the youngest members who diverge most from traditional norms and cultural orientation.
Our findings underscore the need for more understanding of culture-related response bias. In addition, they point out the need for researchers to be aware of the existence and potential distorting effects of response bias in certain cultures that research increasingly indicates are so disposed, such as Latin cultures and Asian cultures such as those found in China, Korea, and Taiwan. Our results point out the need to be cautious in employing surveys that may contain large proportions of Hispanics. Statistical comparisons of American samples should take into account the relative percentage of Hispanics in each sample. In cases, where extreme response is shown, corrective measures--such as normalizing the data through Z scores--should be taken.
Locus of control items
1. Getting the job you want is mostly a matter of luck. 2. Making money is primarily a matter of good fortune. 3. In order to have a good job, you have to have family members or friends in high places. 4. To make a lot of money, you have to know the right people. 5. It takes a lot of luck to be an outstanding employee on most jobs.
6. Employees like to work in a group rather than by themselves. 7. If a group is slowing down, it is better to leave it and work alone. 8. One does better working alone than in a group. 9. Problem-solving by groups gives better results than problem-solving by individuals. 10. An employee should accept the group's decision, even when personally he or she has a different opinion.
Items from Chen et al. (1995) study
11. It is very important to me that I go to college. 12. It is important to my parents that I get good grades. 13. It is important to my parents that I be good at sports. 14. It is important to me that I have many friends. 15. I am good at math. 16. I am good at science. 17. I am above average at athletic ability. 18. I worry about keeping up with my schoolwork. 19. I get nervous when taking a test. 20. I am doing as well in school as I want to. 21. Generally, I am satisfied with myself.
Adler, N. J., Campbell, N., & Laurent, A. (1989). In search of appropriate methodology: From outside the People's Republic of China Looking in. Journal of International Business Studies, Second Quarter, 467-482.
Brislin, R. W. (1970). Back-translation for cross-cultural research. Journal of Cross-Cultural Psychology, 1, 185-216.
Chen, C., Lee, S., & Stevenson, H. W. (1995). Response style and cross-cultural comparisons of rating scales among East Asian and North American students. Psychological Science, 6, 170-175.
Chun, K. T., Campbell, J. B., & Yoo, J. H. (1974). Extreme response style in cross-cultural research: A reminder. Journal of Cross-Cultural Psychology, 5, 465-480.
Cronbach, L. J. (1946). Response sets and test validity. Educational and Psychological Measurement, 6, 475-494.
Cunningham, W. H., Cunningham, I. C., & Green, R. T. (1976). The Ipsative Process to Reduce Response Set Bias Journal of Cross-Cultural Psychology, 7, 225-240.
Hui, C. H., & Triandis, H. C. (1989). Effects of cultural and response format on extreme response style. Journal of Cross-Cultural Psychology, 20, 296-309.
Jaccard, J., & Wan, C. K. (1986). Cross-cultural methods for the study of decision-making. Journal of cross-cultural psychology, 17, 123-149.
Leung, K., & Bond, M. H. (1989). On the empirical identification of dimensions for cross-cultural comparisons. Journal of cross-cultural psychology, 20, 133-151.
Mullen, M. (1995). Diagnosing measurement equivalence in cross-national research. Journal of International Business Research, 573-597.
Singh, J. (1995). Measurement Issues in cross-national research. Journal of International Business Studies, Third Quarter, 597-619.
Stening, B. W., & Everett, J. E. (1984). Response styles in a cross-cultural managerial study. Journal of Social Psychology, 122, 151-156.
Zax, M., & Takashi, S. (1967). Cultural influences on response style: Comparisons of Japanese and American college students. Journal of Social Psychology, 71, 3-10.
Robert A. Culpepper, Stephen F. Austin State University
Raymond A. Zimmerman, University of Texas at El Paso
Table 1: Intercorrelations Among Study Variables Mean S.D. (1) (2) (1) Extreme response score 4.32 3.17 (2) Midpoint response score 1.82 1.88 -.33** (3) Hispanic ethnicity 1.51 .50 .12* -.14* (4) Age 26.74 5.95 -.11* -.03 (5) Sex 1.46 .50 .11 -.09 (6) Dependents 58 1.02 -.10 .00 (3) (4) (5) (1) Extreme response score (2) Midpoint response score (3) Hispanic ethnicity (4) Age .14* (5) Sex .11* -.06 (6) Dependents .01 .62** -.06 ** Correlation is significant at the 0.01 level (2-tailed test). * Correlation is significant at the 0.05 level (2-tailed test). Table 2: MANOVA results Compaing Response Tendencies of Hispanics and Non-Hispanics Hispanics Mean S.D. Multivariate GLM model Extreme response scores 4.73 2.72 > Midpoint response scores 1.58 1.53 < Non-Hispanics Mean S.D. Multivariate GLM model Extreme response scores 3.96 3.45 Midpoint response scores 2.09 2.17 Index Sig. Multivariate GLM model 9.75 (1) .019 Extreme response scores 4.93 (2) .028 Midpoint response scores 5.96 (2) .015 Note: MANOVA assessed differences between cultural groups for the number of extreme (1,7) and midpoint response (4) scores given by each respondent. (1) Wilk's Lambda value for the full model (2) F value for respective difference tests.
|Printer friendly Cite/link Email Feedback|
|Author:||Culpepper, Robert A.; Zimmerman, Raymond A.|
|Publication:||Journal of International Business Research|
|Date:||Jul 1, 2006|
|Previous Article:||The causality between stock index returns and volumes in the Asian equity markets.|
|Next Article:||Convergence in major euro-zone stock markets: evidence from monthly data.|