The impact of question format in principles of economics classes: evidence from New Zealand.
This study investigates whether question format biases assessment results against certain types of students. Ideally, assessment techniques should test subject knowledge and associated skills such as application, analysis and evaluation. They should be blind to student characteristics such as gender and ethnicity, at least to the extent that these are unrelated to academic achievement. An advantage or disadvantage based on an irrelevant characteristic introduces a bias into the results. A better understanding of how question format impacts the relative performance of different types of students can help to avoid these kinds of biases and lead to more reliable assessment techniques.
Universities typically use a mixture of multiple choice (MC) and constructed response (CR) questions in assessing student knowledge in principles of economics classes. In MC questions, the student must choose between a number of possible answers supplied by the assessment. In CR questions, the student is expected to supply the answer. CR questions include fill-in-the blanks, definitions, and short-and long-essays.
Table 1 shows the mix of MC and CR questions employed in introductory economics courses at major universities in New Zealand. Given the higher costs associated with grading CR questions, it is perhaps surprising that CR questions are used as frequently as they are. No university course relies exclusively on MC questions for all their assessments. Some, like the principles of micro- and macroeconomics classes at the University of Auckland, rely entirely on CR questions. Most use a question format that employs a mixture of MC and CR questions. My study intends to shed light on whether certain student groups tend to be relatively disadvantaged by the given mix of MC and CR questions at their university.
A number of studies examine the relationship between performance in university economics classes and student characteristics, with gender being the most frequently studied characteristic. Anderson, Benjamin and Fuss (1994) find that males outperform females overall in introductory economics courses and this advantage persists under several sets of controls. Ziegert (2000) explains this difference using the Myers-Briggs (personality) Type Indicator (MBTI) and finds the gender difference disappears when personality type is controlled for. In particular, for those who are familiar with the MBTI, the statistically higher likelihood of males being T on the T-F scale explains the gender difference. Several studies find that males perform better on MC questions while females perform better on CR (e.g., Lumsden & Scott, 1987). Walstad and Robson (1997) find that the difference in MC between males and females can be reduced by eliminating questions with a clear and identifiable bias, although the full difference is not eliminated.
Race (in the US context) has also attracted study, although most of it has been in terms of overall achievement. Walstad and Robson (1997) find that black students score lower on MC compared to non-black students, but this is consistent with overall achievement. In New Zealand, Juhong and Maloney (2006) find that ethnicity is related to achievement and drop-out rates at the university level. Juhong and Maloney are able to control for level of high school qualification, school decile, (1) gender and age.
This study uses a unique data set to examine the relationship between question format and assessment results for different types of students in university principles-of-economics classes. The data consist of over 20,000 assessment results. Further, they include greater detail about student academic ability than previous studies. I find that student academic ability can explain almost all of the apparent discrepancies in relative performances on MC and CR questions. With one exception, the only student characteristics for which significant differences exist are those that are connected to English language ability. The one exception is gender. I find that female students do relatively better on CR questions, but only in macroeconomics classes.
This paper proceeds as follows. Section 1 introduces the topic and situates this study in the literature. Section 2 describes the data. Section 3 presents and analyses the results. Section 4 concludes.
This study combines assessment data for Principles of Economics courses at the University of Canterbury, New Zealand with demographic data collected by the university. The assessment data covers the period 2002 to 2008 and contains both microeconomics and macroeconomics principles courses. For each student within each course there are two items of assessment--a term test and a final exam. Both assessment items contain MC and CR questions. While the format of the term test and final exam has remained consistent over time, some minor changes in weighting and content have occurred (see Appendix 1). The MC and CR sections for each term test and exam have been scaled to a percentage to allow the two sections to be compared and to allow comparison across different years.
The demographic data are collected by the university at time of enrolment. Self-declared variables include gender, first language and ethnicity. Students are recorded as international or domestic according to their admission criteria. As is typical of most datasets containing self-declared data, the data are somewhat messy. Appendix 2 contains details of how the data were cleaned to provide usable information.
My study focuses on the following student categories:
(1) GENDER--either male or female.
(2) ETHNICITY this variable has five possible categories: European; Maori; Asian; Pacific Island; and Other.
(3) FIRST LANGUAGE--this variable has 3 possible categories: English; Chinese; and Other.
(4) INTERNATIONAL--this variable identifies whether the student pays international (as opposed to domestic) fees.
One issue with these categories is that they can be highly correlated, particularly ethnicity, language and whether or not a student is international or domestic. For example, students who declare their first language to be English are most likely to be domestic students. My detailed data will allow me to identify the independent effects of student characteristics once other characteristics are controlled.
I attempt to control for a student's overall level of ability by calculating a GPA value for each student in the year that they took their first year economics course(s) but that excludes their first year economics courses. This non-economics GPA variable (NEGPA) provides a measure for general student ability.
One of the problems with this variable is that students may take different sets of non-economics classes, and these may have different grade distributions. Accordingly, I make use of the fact that a large number of economics students take a common set of courses. First-year accounting, management, mathematics, and statistics are courses frequently taken by economics students (see Appendix 3). By comparing grades in these common classes, I am able to get a more detailed measure of student ability than previous studies.
The mean values for each of the respective demographic sets are reported in Table 2. The overall picture is clear: females perform below males; non-European ethnicities perform below Europeans; students whose first language is not English perform below students whose first language is English; and international students perform below domestic students. This general picture applies to student performance on both the MC and CR portions of assessments.
I begin by representing a student's performance on the MC and CR portions of assessments as a function of each of the demographic variables in question. Hence, I estimate the following set of equations using ordinary least squares (OLS):
CR = [[alpha].sub.0] + [[alpha].sub.1] (Gender) + [epsilon], (1a)
MC = [[beta].sub.0] + [[beta].sub.1] (Gender) + [epsilon]
CR = [[alpha].sub.0] + [[alpha].sub.1] (Ethnicity) + [epsilon] (1b)
MC = [[beta].sub.0] + [[beta].sub.1] (Ethnicity) + [epsilon]
CR = [[alpha].sub.0] + [[alpha].sub.1] (First Language) + [epsilon], (1c)
MC = [[beta].sub.0] + [[beta].sub.1] (First Language) + [epsilon].
CR = [[alpha].sub.0] + [[alpha].sub.1] (International) + [epsilon], (1d)
MC = [[beta].sub.0] + [[beta].sub.1] International) + [epsilon].
The omitted variables for gender, ethnicity, first language and international are male, European, English and domestic respectively. This first specification allows me to examine the mean score for each assessment type as a function of the respective demographic variable on its own.
Given equation sets (1a) to (1d), I can define the 'relative advantage' (or 'relative disadvantage') a student receives from CR questions by
(CR - MC) = ([[alpha].sub.0] - [[beta].sub.0]) + ([[alpha]].sub.1] - [[beta].sub.1]) (Gender) + [epsilon], (2a)
and so on for equations (2b), (2c) and (2d) respectively.
A positive coefficient indicates that the respective student type is associated with a relative advantage in CR questions. A negative coefficient indicates that the student type experiences a relative disadvantage in CR questions.
These initial results are reported in Table 3 and, so that the reader is able to see how the (CR - MC) coefficient is constructed, shows the coefficients for the CR equations and the MC equations. It is the coefficients for (CR MC) that are of most interest in this study, hence it is those results that are reported in subsequent tables (although for completeness, coefficients for MC and CR are reported in Appendix 4). What Table 3 reveals is that females perform worse in both MC and CR but the disadvantage is significantly greater in Me compared to CR, hence the positive coefficient in the (CR - MC) coefficient column. All ethnicities perform worse compared with the European group in both MC and CR with a greater relative disadvantage in CR. The same is true for students whose first language is not English and for international students compared to domestic.
However, it is possible that what these results reveal is simply student ability or a broader struggle with university study (e.g. Juhong & Maloney, 2006). Thus, to the basic specification, I then add the Academic Ability Variables. These include the non-economics GPA variable (2) discussed above, along with a set of dummy variables and interaction terms to capture student performance in first-year accounting, management, mathematics, and statistics courses. These variables allow me to determine whether any estimated relative advantage or disadvantage persists after controlling for student ability. Hence equation set (la) and (2a) above becomes:
CR = [[alpha].sub.0] + [[alpha].sub.1] (Gender) + [[alpha].sub.2] (Academic Ability Variables) + [epsilon], (3a) and MC = [[beta].sub.0] + [[beta].sub.1] (Gender) + [[beta].sub.2] (Academic Ability Variables) + [epsilon].
(CR - MC) = ([[alpha].sub.0] [[beta].sub.0] + ([[alpha].sub.1] - [[beta].sub.1]) (Gender) + ([[alpha].sub.2] - [[beta].sub.2])(Academic Ability) + [epsilon] (4a)
and so on for the other demographic categories.
Results for the (CR - MC) coefficient, ([[alpha].sub.i] - [[beta].sub.i]), for each type of student grouping are reported in column (2) of Table 4 (Individual Categories + Ability Controls). Column (1) of Table 4 is identical to the third column of Table 3 (Individual Categories).
The final specification combines all of the demographic and academic ability variables into one regression as follows:
CR - [[alpha].sub.0] + [[alpha].sub.1] (Gender, Ethnicity etc) + [[alpha].sub.2] (Academic Ability Variables) + [epsilon], (5)
MC = [[beta].sub.0] + [[beta].sub.1] (Gender, Ethnicity etc) + [[beta].sub.2] (Academic Ability Variables) + [epsilon]. (6)
(CR - MC) = ([[alpha].sub.0] - [[beta].sub.0]) ([[alpha].sub.1] - [[beta].sub.1])(Gender, etc) + ([[alpha].sub.2]) - [[beta].sub.2]) (Academic Ability) + [epsilon].
Column (3) of Table 4 reports coefficients for relative advantage for specification 6 (All Categories + Ability Controls). What do the results in Table 4 reveal?
Gender. As noted above, the positive coefficient for (CR - MC) in column (1) indicates a relative advantage in CR for females compared with males. This is consistent with previous studies. When controls for student ability are introduced into the separate category regression (cf. column 2), the coefficient remains positive but drops below the level of significance (p-value = 0.2309). However, when all the student categories are included in the regression (cf. column 3), the coefficient becomes significant at the 5% level again (p = 0.0125).
Ethnicity. From Table 4, all ethnic groups have a relative disadvantage compared with Europeans in CR as the coefficients are negative. However, this disadvantage disappears for all but the Asian ethnicity group when controls for student quality are introduced. When all the categories and controls for student ability are run together none of the ethnicity groups are significant (column 3 of Table 4). The fact that the coefficient for Asian becomes insignificant when all the categories are run together indicates that rather than an ethnicity issue, this is likely to be a language issue. Not all students who declare Asian as their ethnic group will be second language English speakers. Effectively, this is controlled for when all categories are run together and ethnicity becomes irrelevant.
The overall conclusion is that there is no relative disadvantage in CR for Maori, Pacific Island or Other and the use of CR questions does not discriminate against these groups.
The absolute disadvantage that non-European groups experience in both MC and CR as shown in Table 3 is therefore partly explained by a much broader struggle with University study. Juhong and Maloney (2006) find a very similar result.
So is assessment type blind to ethnicity? Clearly, ethnic groups different to the control group do not perform as well on either MC or CR but the difference is generally not significant when student ability is controlled for. This is a comforting result for instructors who wish to employ both MC and CR questions.
First language. Students with Chinese as their first language clearly have a relative disadvantage in CR. The corresponding coefficient is negative and strongly significant even after controlling for student ability. This latter result is similar to the ethnicity result, which is not surprising given that language and ethnicity are highly collinear. There are 3128 students (3) in the dataset who declare their first language to be Chinese with 3104 in the Asian ethnicity category. Hence, the Chinese category for language is almost entirely contained in the ethnicity category of Asian.
There is some weak evidence that Other language speakers also have a relative disadvantage in CR. The coefficient, when controlling for student ability, is significant at the 10% level when all categories are run together and is only just outside the same level of significance when language is run separately (p = 0.1169).
The implication of these results is that, as would be expected, non-English speaking background students do in fact face a relative disadvantage in CR questions. (4) In the case of language, this is probably reasonable. While not the direct focus of what is being assessed, CR questions will test language ability to a much greater extent than MC. The ability to interpret more open ended questions, formulate ideas and then communicate those ideas is an important skill in economics. In the end, students who graduate from the University of Canterbury do so with a degree from an English-speaking university, so being able to read, interpret and communicate in English is important. Students for whom English is not their first language and who plan to enter an English-speaking university may well find that improving their English ability prior to entry is the single most important thing they can do to improve chances of success.
International. International students appear to have a relative disadvantage in CR in the absence of the other categories. However, this is likely to be picking up the effect of language. When language, gender and ethnicity are included (specification 6) the coefficients in Table 4 become insignificant (a result that is also true even without the controls for student ability included). Being an international student per se is not likely to result in a relative disadvantage in either MC or CR.
This is not only because most of the effect is captured by language but also because not all international students have English as a second language. Of the 5902 international students, 658 declare English as their first language and 790 declare Other.
Macroeconomics and microeconomics. Table 5 shows the same results as column (3) of Table 4 but divided into microeconomics and macroeconomics. The macroeconomics results are the same as the overall results in Table 4, while the microeconomics results differ somewhat. Asian ethnicity shows an advantage in CR for microeconomics. One possible explanation is that microeconomics questions have a greater graphical, algebraic and computational element than questions in macroeconomics, which require more language skills. This may well play into the hands of students with a stronger mathematical facility, including students of Asian ethnicity. This could be consistent with the observation for females where the advantage that females have in CR disappears for microeconomics. The relative advantage for females may arise, at least in part, from their relatively better language performance. The greater premium on symbolic representation and manipulation in microeconomics would then reduce this relative advantage. (5)
This study investigates whether some types of students are unfairly disadvantaged by the question format on assessments in university economics classes. I find that in the absence of controls for student academic ability, question format appears to be biased against many types of students. However, once academic ability is accounted for, most of the estimated effects become statistically insignificant.
Where significant effects remain, these can almost always be related to English language ability. Students who do not have English as their first language have a relative disadvantage in constructed response (CR) questions compared to students with English as their first language. While course instructors are not usually aiming to test language directly, language is an important component of academic study. Students need to be able to absorb and interpret information presented in English as well as form their own ideas and express these clearly. Multiple choice (MC) questions do not test such skills to the same extent as constructed response. As such, it can be argued that the relative disadvantage experienced by non-native English speakers on CR questions reflects a valid dimension of student achievement.
The only non-language related student characteristic that remained significant after controlling for student ability was gender. I find that female students do relatively better on CR questions, but only in macroeconomics classes. This result may be of interest to instructors of principles of macroeconomics courses. The use of all-MC tests and exams will disadvantage female students in a discipline where females already perform below males.
This study raises a number of interesting questions. Why do females continue to score less than males in economics when their broader GPA is in fact higher? (6) What classroom strategies might be put in place to overcome this? What kind of language-training can help non-native English speakers to reduce the disadvantage they face on CR questions? These questions remain for future research.
Appendix 1. How the test and exam have changed over time
2002 and 2003, semester 1: Assignment 10%, term test 40%, final exam 50%. Final exam: three constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 50/100, CR worth 50/100.
2003 semester 2 and 2004: Assignment 10%, term test 40%, final exam 50%. Final exam: two constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 50/100, CR worth 50/100.
2005 and 2006: Assignment 10%, online MC 10%, term test 35%, final exam 45%. Final exam: two constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 25/75, CR worth 50/75.
2007 onwards: Assignment 10%, online MC 10%, term test 20%, final exam 60%. Final exam: two constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 25/75, CR worth 50/75.
Note that from 2007 onwards students needed to pass the final exam or get 39/80 in the test and exam combined to get a full pass. The final exam became comprehensive and included a greater coverage of term 1 material than before.
The second semester offering of microeconomics was introduced in 2003. The first semester offering of macroeconomics was introduced in 2006.
Appendix 2. Details about the construction of the demographic variables
The student management system at the University of Canterbury collects data on a range of student characteristics. Students self-report their characteristics for each year they are enrolled. Some characteristics are not compulsory to complete and so may have missing values, e.g. ethnicity. Despite this, the database provides a rich source of information with which to classify students.
A complication arises because some students take their introductory economics courses over multiple years. Reasons for this include the fact that students may choose to spread out their study, or because they fail a course. In these cases, the student management system contains multiple records, one for each year the student was enrolled in an introductory economics course. Because of the self-declared nature of the data and the fact that some fields legitimately change over time (e.g., a student may gain NZ citizenship while studying), the same student may look different from one year to the next. A 'best judgment' was used to determine the most appropriate classifications for these students. If this could not be done with reasonable certitude, the student was dropped from the sample.
Citizenship. There are three values for citizenship in the student database: (i) New Zealand, (ii) Permanent Resident, and (iii) Overseas. Permanent residents are students who do not have citizenship but are entitled to reside in New Zealand on an ongoing basis.
Ethnicity. The student database contains a primary ethnicity field that supplies the following choices: (i) European, (ii) Maori, (iii) Pacific Islander, (iv) Asian, (v) Indian, and (vi) Other. Most students self-report one of these categories. However, students have the opportunity to supply their own ethnic identification. In addition, they can select multiple ethnicities. The upshot is that the student management system assigns a value of 'Unknown' for many students' ethnicities. Nevertheless, most students were able to be assigned to an ethnicity category using other information such as citizenship type and country of citizenship.
'Indian' was eliminated as a separate ethnicity category. This category accounted for less than 50 students in the full sample. In some of the subsamples used in the empirical analyses, there were no students in this category. As a result, Indian was included in Other.
The table below summarizes how student were assigned to the ethnic categories used in this study.
Assigned ethnic category Ethnicity reported in student records European -NZ European/European/Pakeha -NZ European/Pakeha -Australian -Unknown' and country of citizenship is any of New Zealand, United Kingdom, France, United States, Netherlands, Latvia, Russia, Ukraine, Australia, Norway, Canada or Germany. Asian -Chinese -Filipino -Other Asian -Unknown' and country of citizenship is any of China, South Korea, Malaysia, Thailand, Japan, Philippines, Indonesia, Pakistan, Sri Lanka or Singapore. Maori -New Zealand Maori Pacific Islander -Fijian -Samoan -Tongan Other -Unknown' and country of citizenship is any of Nigeria, Zimbabwe, South Africa, Kenya, Tanzania, Sudan, Maldives Islands, Ethiopia, Egypt, Saudi Arabia or Chile.
Language. The 'First Language' field in the student information file supplies the following categories: (i) English; (ii) Mandarin, (iii) Other Chinese Dialect, (iv) Other Asian, (v) Maori, (vi) Other, and (vii) Not Specified. As would be expected with self-reported data of this sort, the data are noisy. For example, a student from Hong Kong declared his language as Other Chinese Dialect in one year, but later identified English as his first language. Similarly, a student from Taiwan originally declared Mandarin as his first language, but reported Not Specified a later year. In many cases, these ambiguities are legitimate as many students are highly fluent in more than one language, so that there is little basis for choosing one language as 'first language' over another. Finally, Maori was included with English because there were only three students in the sample who declared Maori as their first language. All of these would be fluent in English. The table below summarizes the language categorization system used for this study.
Assigned language category 'First language' reported in student records Chinese --Mandarin --Other Chinese Dialect --(i) Language reported as 'Not Specified', 'Other' or 'Other Asian;' and (ii) Citizenship = 'Overseas' and Country = 'China' English --English --Maori --(i) Language reported as 'Not Specified', 'Other' or 'Other Asian;' and (ii) Citizenship = 'New Zealand' and Ethnicity = 'European,' OR Citizenship = 'New Zealand' and Ethnicity = 'Maori,' OR Citizenship = 'New Zealand' or 'United Kingdom' or 'United States' or 'Canada' Other --(i) Language reported as 'Not Specified', 'Other' or 'Other Asian;' and (ii) does not meet any of the conditions above Appendix 3. Simple counts of commonly taken courses and combinations of those courses Course combinations Number Percent Individual Courses Accounting 3226 51 Mathematics 2298 36 Statistics 4184 66 Management 3798 60 Combinations of two courses Accountancy and Mathematics 1364 22 Accountancy and Statistics 2746 43 Accountancy and Management 2366 37 Mathematics and Statistics 1810 29 Mathematics and Management 1187 19 Statistics and Management 3008 48 Combinations of three courses Accountancy, Mathematics and Statistics 1249 20 Accountancy, Mathematics and Management 869 14 Accountancy, Statistics and Management 2108 33 Mathematics, Statistics and Management 1078 17 All four courses 823 13 Taken none of the four courses 792 13 Total number of individual students 6313 100 Appendix 4. Coefficients for CR and MC separately Gender CR No controls With controls Constant 52.3 *** (264.93) 33.37 *** (78.63) Female -0.84 *** (-2.81) -2.68 *** (-12.01) Non Econ GPA 4.79 *** (64.38) Accy. -1.34 *** (-3.76) Accy GPA 0.70 *** (12.12) Math. 2.63 *** (7.54) Math GPA -0.03 (-0.43) Stat. -0.01 (-0.02) Stat GPA 0.44 *** (7.28) Mgmt. -3.44 *** (-9.52) Mgmt GPA 0.66 *** (10.24) Observations 20446 20254 [R.sup.2] 0.0004 0.4590 MC No controls With controls Constant 68.9 *** (486.37) 58.0 *** (167.37) Female -1.93 *** (-8.91) -2.94 *** (-16.42) Non Econ GPA 2.91 *** (48.55) Accy. -1.54 *** (-5.19) Accy GPA 0.47 *** (9.89) Math. 1.38 *** (4.76) Math GPA 0.06 (1.07) Stat. 1.13 *** (3.69) Stat GPA -0.03 (-0.71) Mgmt. -3.96 *** (-13.06) Mgmt GPA 0.77 *** (14.57) Observations 20446 20254 [R.sup.2] 0.0039 0.3376 Note: Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the 1 % level, ** at the 5% level and * at the 10% level. Ethnicity CR No controls With controls Constant 54.99 *** (275.72) 33.73 *** (78.78) Maori Ethnicity -5.87 *** (-6.38) -1.58 ** (-2.25) Asian Ethnicity -6.28 *** (-20.40) -3.93 *** (-14.64) Pacific Island -14.20 *** (-13.22) -2.71 *** (-3.05) Other Ethnicity -6.53 *** (-7.94) -2.04 *** (-3.47) Non Econ GPA 4.63 *** (61.40) Accy. -0.90 ** (-2.53) Accy GPA 0.69 *** (11.99) Math. 3.54 *** (10.20) Math GPA 0.06 (0.91) Stat. 0.19 (0.51) Stat GPA 0.51 *** (8.47) Mgmt. -3.50 *** (-9.74) Mgmt GPA 0.55 *** (8.42) Observations 20446 20254 [R.sup.2] 0.0260 0.4610 MC No controls With controls Constant 70.30 *** (496.4) 57.92 *** (165.37) Maori Ethnicity -4.28 *** (-6.28) -1.61 *** (-2.68) Asian Ethnicity -4.50 *** (-20.06) -2.61 *** (-12.14) Pacific Island -10.06 *** (-11.75) -2.92 *** (-3.87) Other Ethnicity -5.00 *** (-8.41) -2.21 *** (-4.78) Non Econ GPA 2.80 *** (46.21) Accy. -1.31 *** (-4.42) Accy GPA 0.46 *** (9.68) Math. 2.10 *** (7.27) Math GPA 0.09 * (1.73) Stat. 1.30 *** (4.23) Stat GPA 0.00 (0.04) Mgmt. -4.08 *** (-13.45) Mgmt GPA 0.69 *** (12.89) Observations 20446 20254 [R.sup.2] 0.0256 0.3341 Note: Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the 1% level, ** at the 5% level and * at the 10% level. Language CR No controls With controls Constant 53.90 *** (295.88) 33.09 *** (78.79) Chinese language -5.75 *** (-17.23) -4.54 *** (-15.42) Other language -5.84 *** (-10.41) -2.15 *** (-4.89) Non Econ GPA 4.66 *** (62.16) Acc -0.72 ** (-2.03) Accy. GPA 0.68 *** (11.77) Math. 3.56 *** (10.26) Math GPA 0.11 * (1.69) Stat. 0.28 (0.74) Stat GPA 0.50 *** (8.35) Mgmt. -3.39 *** (-9.43) Mgmt GPA 0.53 *** (8.15) Observations 20446 20254 [R.sup.2] 0.0169 0.4614 MC No controls With controls Constant 69.37 *** (530.07) 57.29 *** (165.87) Chinese language -3.62 *** (-14.77) -2.11 *** (-8.71) Other language -4.06 *** (-9.72) -1.49 *** (-4.22) Non Econ GPA 2.84 *** (47.10) Acc -1.34 *** (-4.49) Accy. GPA 0.45 *** (9.50) Math. 2.04 *** (7.02) Math GPA 0.09 * (1.70) Stat. 1.31 *** (4.26) Stat GPA -0.02 (-0.37) Mgmt. -4.04 *** (-13.26) Mgmt GPA 0.70 *** (13.08) Observations 20446 20254 [R.sup.2] 0.0134 0.3315 Note: Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the 1 % level, ** at the 5% level and * at the 10% level. International versus domestic students CR No controls With controls Constant 53.48 *** (302.83) 33.02 *** (78.62) International -5.50 *** (17.31) -3.64 *** (-13.11) Non Econ GPA 4.70 *** (62.64) Accy. -0.96 *** (-2.69) Accy GPA 0.66 *** (11.59) Math. 3.40 *** (9.79) Math GPA 0.09 (1.38) Stat. 0.20 (0.54) Stat GPA 0.46 *** (7.72) Mgmt. -3.46 *** (-9.60) Mgmt GPA 0.55 *** (8.50) Observations 20446 20254 [R.sup.2] 0.0139 0.4598 MC No controls With controls Constant 69.13 *** (545.02) 57.26 *** (165.71) International -3.65 *** (-15.57) -2.02 *** (-8.97) Non Econ GPA 2.86 *** (47.36) Accy. -1.41 *** (-4.73) Accy GPA 0.45 *** (9.37) Math. 2.00 *** (6.91) Math GPA 0.09 * (1.81) Stat. 1.29 *** (4.20) Stat GPA -0.03 (-0.68) Mgmt. -4.05 *** (-13.31) Mgmt GPA 0.70 *** (13.15) Observations 20446 20254 [R.sup.2] 0.0116 0.3315 Note: Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the 1 % level, ** at the 5% level and * at the 10% level. All demographic variables CR No controls With controls Constant 55.1 *** (239.17) 34.33 *** (78.88) Female -0.07 (-0.25) -2.14 *** (-9.47) Maori Ethnicity -5.93 *** (-6.44) -1.48 ** (-2.10) Asian Ethnicity -3.47 *** (-6.06) -1.67 *** (-3.86) Pacific Island -13.55 *** (-12.48) -2.13 ** (-2.41) Other Ethnicity -5.24 *** (-6.07) -1.38 ** (-2.25) Chinese language -2.02 *** (-3.08) -2.10 *** (-4.23) Other language -2.43 *** (-3.52) -0.52 (-0.98) International -1.68 *** (-3.19) -1.07 *** (-2.74) Non Econ GPA 4.63 *** (61.54) Accy. -0.61 * (-1.71) Accy GPA 0.69 *** (11.98) Math. 3.29 *** (9.41) Math GPA 0.15 ** (2.32) Stat. 0.19 (0.50) Stat GPA 0.52 *** (8.76) Mgmt. -3.31 *** (-9.24) Mgmt GPA 0.53 *** (8.18) Observations 20446 20254 [R.sup.2] 0.0280 0.4650 MC No controls With controls Constant 70.90 *** (437.90) 58.80 *** (166.71) Female -1.47 *** (-6.76) -2.70 *** (-14.85) Maori Ethnicity -4.24 *** (-6.21) -1.45 ** (-2.40) Asian Ethnicity -3.59 *** (-8.47) -2.21 *** (-6.34) Pacific Island -9.63 *** (-11.20) -2.46 *** (-3.26) Other Ethnicity -4.34 *** (-6.95) -1.97 *** (-4.10) Chinese language 0.16 (0.33) 0.56 (1.40) Other language -0.90 * (-1.76) 0.36 (0.86) International -1.11 *** (-2.88) -0.63 ** (-1.98) Non Econ GPA -2.81 *** (46.62) Accy. -1.22 *** (-4.11) Accy GPA 0.47 *** (9.91) Math. 1.69 *** (5.82) Math GPA 0.13 ** (2.47) Stat. 1.20 *** (3.93) Stat GPA 0.01 (0.27) Mgmt. -3.94 *** (-13.15) Mgmt GPA 0.71 *** (13.31) Observations 20446 20254 [R.sup.2] 0.0285 0.3415 Note: Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the 1 % level, ** at the 5% level and * at the 10% level.
The author wishes to acknowledge Bob Reed for his support and extremely helpful comments with this paper.
Anderson, G., Benjamin, D., & Fuss, M.A. (1994). The determinants of success in university introductory economics courses. Journal of Economic Education, 25(2), 99 119.
Juhong, B., & Maloney, T. (2006). Ethnicity and academic success at university. New Zealand Economics Papers, 40(2), 181-213.
Lumsden, K.G., & Scott, A. (1987). The economics student reexamined: Male-female differences in comprehension. Journal of Economic Education, 18(4), 365 375.
Walstad, W.B., & Robson, D. (1997). Differential item functioning and male-female differences on multiple-choice tests in economics. Journal of Economic Education, 28(2), 155 171.
Ziegert, A.L. (2000). The role of personality temperament and student learning in principles of economics: Further evidence. Journal of Economic Education, 3l(4), 307-322.
(1.) Broadly speaking, school decile is a measure of the overall affluence of the school that the student attends. The New Zealand Ministry of Education website (www.minedu.govt.nz) has the following statement:
'A school's Decile indicates the extent to which it draws its students from low socioeconomic communities. Decile 1 schools are the 10% of schools with the highest proportion of students from low socio-economic communities. Decile 10 schools are the 10% of schools with the lowest proportion of these students.'
(2.) One possible difficulty with using a variable like NEGPA is that this may, like the dependent variable under study, be prone to error due to the omission of an underlying and probably unobservable ability variable, which means that the error term is likely to be correlated with the NEGPA variable. This brings the usual problems for the OLS estimates. However, the primary effect of unobservable ability should be minimized via the differencing of CR and MC scores. If I rewrite equations (1a) as follows:
[CR.sub.i] = [[alpha].sub.0] + [[alpha].sub.1] [Gender.sub.i] [[lambda].sub.i] + v
[MC.sub.i] = [[beta].sub.0] + [[beta].sub.1] [Gender.sub.i] + [[lambda].sub.i] + [mu]
where [lambda]i is the unobservable ability variable. Then
([CR.sub.i] - [MC.sub.i]) = ([[alpha].sub.0] - [[beta].sub.0]) + ([[alpha].sub.1] -/ [[beta].sub.1]) [Gender.sub.i] + (v - [mu]).
The preceding assumes that ability has the same effect on CR and MC scores. If, instead, ability has a differential effect on the two scores, then the corresponding difference equation is
([CR.sub.i] [MC.sub.i]) - ([[alpha].sub.0] - [[beta].sub.0]) + ([[alpha].sub.1] - [[beta].sub.1]) [Gender.sub.i] + ([[lambda].sub.i,CR] - [[lambda].sub.i,MC]) [Ability.sub.i] + (v - [mu]).
If my ability controls do not control for all relevant ability influences, and these unobserved ability influences are correlated with my controls (such as NEGPA), then it is correct that the error term will likely be correlated with the included ability controls. This will generate biased estimates of the ability control coefficients. However, as long as these unobservable ability influences are uncorrelated with the gender/ethnicity/etc, variables, these latter coefficients--which are the main objects of my analysis should be unbiased.
(3.) This number is calculated by treating a student who appears in multiple years as a different student in each year since self-declared demographic characteristics can change from year to year.
(4.) While this result holds over all the data it may not hold at every point in time or in every particular course offering. The main exception is the 2004 year where the relative disadvantage is in MC rather than CR for Chinese language speakers. In 2003, the percentage of Chinese language students climbed to 40% of the Microeconomics class as the number of Chinese students entering NZ education rose dramatically. It is possible that, consciously or otherwise, an over-compensation was made by the instructor in the following year in response to this increase in either question setting or marking. Another possibility is that the students who arrived in this 'bubble' were qualitatively different to the long-term average Chinese speaking student. The exact cause is difficult to determine and it may simply be 'noise' in the data.
(5.) Confounding this somewhat is that females in the sample who have taken the common first year mathematics paper have a mean GPA of 4.4 compared with the mean for males of 3.5 in the mathematics course. Nevertheless, it can still be the case that the lower emphasis on language skills by virtue of the presence of an emphasis on mathematical-type skills is sufficient to remove the advantage in CR.
(6.) The mean value for NEGPA for females is 3.8 (s.e. = 0.038) but it is only 3.5 (s.e. = 0.033) for males. GPA is measured on a scale of-I to 9 where a grade of E has a GPA value of 1, D is 0, C- is 1 and so on up to an A+ which has a GPA value of 9.
Stephen Hickson *
Department of Economics and Finance, University of Canterbury, Private Bag 4800, Christchurch 8042, New Zealand
(Received 23 March 2010; final version received 28 July 2010)
* Email: firstname.lastname@example.org
Table 1. How New Zealand universities assess principles of economics courses. The following information on invigilated assessment was obtained via an email survey. It was correct at December 2009 but may have subsequently changed. Term test Final University Paper (S) (% MC) exam Auckland (1) Microeconomics 0 0 Macroeconomics 0 0 General Economics 60 30 BBIM General Economics 0 0 AUT (2) Economics 0 Waikato Business Economics and the NZ 60 60 Economy Economics and Society 0 0 Massey Microeconomics 0 100 Macroeconomics 0 100 Victoria Microeconomics 100 70 Macroeconomics 100 70 Lincoln Introduction to Applied Economics 40/50 30 Introduction to Economic Theory 50/65 30 Canterbury Microeconomics 33 30 Macroeconomics 33 30 Otago Principles of Economics 1 60 67 Principles of Economics 11 50 50 Notes: (1) University of Auckland The two general economics papers are not taken by economics majors. Economics majors take ECON 101 Microeconomics and ECON 111 Macroeconomics. BBIM stands for Bachelor of Business Information Management. (2) Auckland University of Technology AUT operate a different teaching model with smaller classes. They have four assessments within the economics section, namely: an online assignment worth 10%; an invigilated test worth 20% (all CR); a portfolio worth 30%; and an essay worth 40%. Table 2. Summary statistics by student category. Category Percent of Term test Final exam sample CR percent CR percent Gender Female 43.3 48.8 54.0 Male 56.7 50.1 54.4 Ethnicity Asian 39.1 46.5 50.9 European 52.8 52.5 57.4 Maori 2.5 46.8 51.4 Pacific Island 1.6 38.4 43.1 Other 4.0 45.6 51.3 First Language Chinese 26.7 45.8 50.5 English 65.3 51.6 56.2 Other 8.0 45.4 50.8 International Domestic 71.1 51.1 55.9 International 28.9 45.8 50.2 Category Term test Final exam MC percent MC percent Gender Female 65.7 68.2 Male 67.6 70.2 Ethnicity Asian 65.1 66.5 European 68.6 72.0 Maori 64.0 68.1 Pacific Island 58.8 61.7 Other 63.3 67.3 First Language Chinese 65.0 66.5 English 67.8 70.9 Other 64.6 66.1 International Domestic 67.5 70.7 International 64.9 66.1 Note: Term Test and Final Exam Marks range from 0 to 100. Table 3. Estimation of coefficients for CR and MC and relative advantage (CR--MC) for separate demographic variables. (1. CR) Coefficients (1. MC) Less for CR coefficients for MC ([[alpha].sub.1]) ([[beta].sub.1]) Gender (a) Female -0.84 *** (-2.81) -1.93 *** (-8.91) Ethnicity (b) Maori -5.87 *** (-6.38) -4.28 *** (-6.28) Asian -6.28 *** (-20.40) -4.50 *** (-20.06) Pacific Island -14.20 *** (-13.22) -10.06 *** (-11.75) Other -6.53 *** (-7.94) -5.00 *** (-8.41) First Language (c) Chinese -5.75 *** (-17.23) -3.62 *** (-14.77) Other -5.84 *** (-10.41) -4.06 *** (-9.72) International (d) International -5.50 *** (-17.31) -3.65 *** (-15.57) Observations 20,446 20,446 (2) Equals relative advantage ([[alpha].sub.1]- [[beta].sub.1]) Gender (a) Female 1.09 *** (4.68) Ethnicity (b) Maori -1.59 ** (-2.13) Asian -1.78 *** (-7.24) Pacific Island -4.14 *** (-4.86) Other -1.54 *** (-2.61) First Language (c) Chinese -2.13 *** (-7.88) Other -1.78 *** (-4.06) International (d) International -1.85 *** (-7.12) Observations 20,446 Notes: (1) The three coefficients in Panel a (Gender) correspond to [[alpha].sub.1], [[beta].sub.1], and ([[alpha].sub.1]-[[beta].sub.1]) in specifications (1a) and (2a) respectively. Panels b-d are interpreted similarly. (2) Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the l % level, ** at the 5% level and * at the 10% level. Table 4. Estimation of the relative advantage associated with CR questions for specific student groups. Individual Individual categories + categories (1) Ability controls (2) Gender (a) Female 1.09 *** (4.68) 0.27 (1.20) Ethnicity (b) Maori -1.59 ** (-2.13) 0.03 (0.04) Asian -1.78 *** (-7.24) -1.32 *** (-5.02) Pacific Island -4.14 *** (-4.86) 0.21 (0.25) Other -1.54 *** (-2.61) 0.17 (0.31) First Language (c) Chinese -2.13 *** (-7.88) -2.43 *** (-8.11) Other -1.78 *** (-4.06) -0.65 (-1.54) International (d) International -1.85 *** (-7.12) -1.62 *** (-5.78) Observations 20,446 20,254 All categories + Ability controls (3) Gender (a) Female 0.57 *** (2.50) Ethnicity (b) Maori -0.03 (-0.05) Asian 0.54 (1.33) Pacific Island 0.33 (0.39) Other 0.59 (1.02) First Language (c) Chinese -2.66 *** (-5.54) Other -0.88 * (-1.75) International (d) International -0.44 (-1.15) Observations 20,254 Notes: (1) Column (1) reports the estimated values of ([[alpha].sub.1]-[[beta].sub.1]) in specifications (2a)-(2d) in the text. Column (2) reports the same after controlling for Ability variables (see specifications (4a)-(4d) in the text). Column (3) reports estimates of (([[alpha].sub.1]-[[beta].sub.1]) corresponding to specification (6) in the text. (2) Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the 1 % level, ** at the 5% level and * at the 10% level. See the text for the description of ability control variables. Table 5. Estimation of the relative advantage associated with CR questions for specific student groups: macro and micro. All Categories + Ability Controls Macroeconomics Microeconomics Female 1.16 *** (3.44) 0.30 (0.99) Maori Ethnicity -0.49 (-0.45) 0.24 (0.26) Asian Ethnicity -0.92 (-1.49) 1.65 *** (3.03) Pacific Island -1.75 (-1.25) 1.83 * (1.70) Other Ethnicity -0.49 (-0.56) 1.38 * (1.79) Chinese language -1.64 ** (-2.26) -3.39 *** (-5.29) Other language 0.61 (0.82) -1.97 *** (-2.91) International 0.34 (0.59) -1.11 ** (-2.14) Observations 8904 11350 Notes: (1) Estimates in each column correspond to ([[alpha].sub.1]-[[beta].sub.1]) from specification (6) in the text. (2) Values in parentheses are t-statistics calculated using heteroscedastic-robust (White) standard errors. *** indicates significant difference from zero at the 1 % level, ** at the 5% level and * at the 10% level. See the text for the description of ability control variables.
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||RESEARCH ARTICLE|
|Publication:||New Zealand Economic Papers|
|Date:||Dec 1, 2010|
|Previous Article:||Choice experiment assessment of public expenditure preferences.|
|Next Article:||The spirit level: Why more equal societies almost always do better.|