Printer Friendly

The impact of question format in principles of economics classes: evidence from New Zealand.

1. Introduction

This study investigates whether question format biases assessment results against certain types of students. Ideally, assessment techniques should test subject knowledge and associated skills such as application, analysis and evaluation. They should be blind to student characteristics such as gender and ethnicity, at least to the extent that these are unrelated to academic achievement. An advantage or disadvantage based on an irrelevant characteristic introduces a bias into the results. A better understanding of how question format impacts the relative performance of different types of students can help to avoid these kinds of biases and lead to more reliable assessment techniques.

Universities typically use a mixture of multiple choice (MC) and constructed response (CR) questions in assessing student knowledge in principles of economics classes. In MC questions, the student must choose between a number of possible answers supplied by the assessment. In CR questions, the student is expected to supply the answer. CR questions include fill-in-the blanks, definitions, and short-and long-essays.

Table 1 shows the mix of MC and CR questions employed in introductory economics courses at major universities in New Zealand. Given the higher costs associated with grading CR questions, it is perhaps surprising that CR questions are used as frequently as they are. No university course relies exclusively on MC questions for all their assessments. Some, like the principles of micro- and macroeconomics classes at the University of Auckland, rely entirely on CR questions. Most use a question format that employs a mixture of MC and CR questions. My study intends to shed light on whether certain student groups tend to be relatively disadvantaged by the given mix of MC and CR questions at their university.

A number of studies examine the relationship between performance in university economics classes and student characteristics, with gender being the most frequently studied characteristic. Anderson, Benjamin and Fuss (1994) find that males outperform females overall in introductory economics courses and this advantage persists under several sets of controls. Ziegert (2000) explains this difference using the Myers-Briggs (personality) Type Indicator (MBTI) and finds the gender difference disappears when personality type is controlled for. In particular, for those who are familiar with the MBTI, the statistically higher likelihood of males being T on the T-F scale explains the gender difference. Several studies find that males perform better on MC questions while females perform better on CR (e.g., Lumsden & Scott, 1987). Walstad and Robson (1997) find that the difference in MC between males and females can be reduced by eliminating questions with a clear and identifiable bias, although the full difference is not eliminated.

Race (in the US context) has also attracted study, although most of it has been in terms of overall achievement. Walstad and Robson (1997) find that black students score lower on MC compared to non-black students, but this is consistent with overall achievement. In New Zealand, Juhong and Maloney (2006) find that ethnicity is related to achievement and drop-out rates at the university level. Juhong and Maloney are able to control for level of high school qualification, school decile, (1) gender and age.

This study uses a unique data set to examine the relationship between question format and assessment results for different types of students in university principles-of-economics classes. The data consist of over 20,000 assessment results. Further, they include greater detail about student academic ability than previous studies. I find that student academic ability can explain almost all of the apparent discrepancies in relative performances on MC and CR questions. With one exception, the only student characteristics for which significant differences exist are those that are connected to English language ability. The one exception is gender. I find that female students do relatively better on CR questions, but only in macroeconomics classes.

This paper proceeds as follows. Section 1 introduces the topic and situates this study in the literature. Section 2 describes the data. Section 3 presents and analyses the results. Section 4 concludes.

2. Data

This study combines assessment data for Principles of Economics courses at the University of Canterbury, New Zealand with demographic data collected by the university. The assessment data covers the period 2002 to 2008 and contains both microeconomics and macroeconomics principles courses. For each student within each course there are two items of assessment--a term test and a final exam. Both assessment items contain MC and CR questions. While the format of the term test and final exam has remained consistent over time, some minor changes in weighting and content have occurred (see Appendix 1). The MC and CR sections for each term test and exam have been scaled to a percentage to allow the two sections to be compared and to allow comparison across different years.

The demographic data are collected by the university at time of enrolment. Self-declared variables include gender, first language and ethnicity. Students are recorded as international or domestic according to their admission criteria. As is typical of most datasets containing self-declared data, the data are somewhat messy. Appendix 2 contains details of how the data were cleaned to provide usable information.

My study focuses on the following student categories:

(1) GENDER--either male or female.

(2) ETHNICITY this variable has five possible categories: European; Maori; Asian; Pacific Island; and Other.

(3) FIRST LANGUAGE--this variable has 3 possible categories: English; Chinese; and Other.

(4) INTERNATIONAL--this variable identifies whether the student pays international (as opposed to domestic) fees.

One issue with these categories is that they can be highly correlated, particularly ethnicity, language and whether or not a student is international or domestic. For example, students who declare their first language to be English are most likely to be domestic students. My detailed data will allow me to identify the independent effects of student characteristics once other characteristics are controlled.

I attempt to control for a student's overall level of ability by calculating a GPA value for each student in the year that they took their first year economics course(s) but that excludes their first year economics courses. This non-economics GPA variable (NEGPA) provides a measure for general student ability.

One of the problems with this variable is that students may take different sets of non-economics classes, and these may have different grade distributions. Accordingly, I make use of the fact that a large number of economics students take a common set of courses. First-year accounting, management, mathematics, and statistics are courses frequently taken by economics students (see Appendix 3). By comparing grades in these common classes, I am able to get a more detailed measure of student ability than previous studies.

3. Results

The mean values for each of the respective demographic sets are reported in Table 2. The overall picture is clear: females perform below males; non-European ethnicities perform below Europeans; students whose first language is not English perform below students whose first language is English; and international students perform below domestic students. This general picture applies to student performance on both the MC and CR portions of assessments.

I begin by representing a student's performance on the MC and CR portions of assessments as a function of each of the demographic variables in question. Hence, I estimate the following set of equations using ordinary least squares (OLS):

CR = [[alpha].sub.0] + [[alpha].sub.1] (Gender) + [epsilon], (1a)

and

MC = [[beta].sub.0] + [[beta].sub.1] (Gender) + [epsilon]

CR = [[alpha].sub.0] + [[alpha].sub.1] (Ethnicity) + [epsilon] (1b)

and

MC = [[beta].sub.0] + [[beta].sub.1] (Ethnicity) + [epsilon]

CR = [[alpha].sub.0] + [[alpha].sub.1] (First Language) + [epsilon], (1c)

and

MC = [[beta].sub.0] + [[beta].sub.1] (First Language) + [epsilon].

CR = [[alpha].sub.0] + [[alpha].sub.1] (International) + [epsilon], (1d)

and

MC = [[beta].sub.0] + [[beta].sub.1] International) + [epsilon].

The omitted variables for gender, ethnicity, first language and international are male, European, English and domestic respectively. This first specification allows me to examine the mean score for each assessment type as a function of the respective demographic variable on its own.

Given equation sets (1a) to (1d), I can define the 'relative advantage' (or 'relative disadvantage') a student receives from CR questions by

(CR - MC) = ([[alpha].sub.0] - [[beta].sub.0]) + ([[alpha]].sub.1] - [[beta].sub.1]) (Gender) + [epsilon], (2a)

and so on for equations (2b), (2c) and (2d) respectively.

A positive coefficient indicates that the respective student type is associated with a relative advantage in CR questions. A negative coefficient indicates that the student type experiences a relative disadvantage in CR questions.

These initial results are reported in Table 3 and, so that the reader is able to see how the (CR - MC) coefficient is constructed, shows the coefficients for the CR equations and the MC equations. It is the coefficients for (CR MC) that are of most interest in this study, hence it is those results that are reported in subsequent tables (although for completeness, coefficients for MC and CR are reported in Appendix 4). What Table 3 reveals is that females perform worse in both MC and CR but the disadvantage is significantly greater in Me compared to CR, hence the positive coefficient in the (CR - MC) coefficient column. All ethnicities perform worse compared with the European group in both MC and CR with a greater relative disadvantage in CR. The same is true for students whose first language is not English and for international students compared to domestic.

However, it is possible that what these results reveal is simply student ability or a broader struggle with university study (e.g. Juhong & Maloney, 2006). Thus, to the basic specification, I then add the Academic Ability Variables. These include the non-economics GPA variable (2) discussed above, along with a set of dummy variables and interaction terms to capture student performance in first-year accounting, management, mathematics, and statistics courses. These variables allow me to determine whether any estimated relative advantage or disadvantage persists after controlling for student ability. Hence equation set (la) and (2a) above becomes:

CR = [[alpha].sub.0] + [[alpha].sub.1] (Gender) + [[alpha].sub.2] (Academic Ability Variables) + [epsilon], (3a) and MC = [[beta].sub.0] + [[beta].sub.1] (Gender) + [[beta].sub.2] (Academic Ability Variables) + [epsilon].

(CR - MC) = ([[alpha].sub.0] [[beta].sub.0] + ([[alpha].sub.1] - [[beta].sub.1]) (Gender) + ([[alpha].sub.2] - [[beta].sub.2])(Academic Ability) + [epsilon] (4a)

and so on for the other demographic categories.

Results for the (CR - MC) coefficient, ([[alpha].sub.i] - [[beta].sub.i]), for each type of student grouping are reported in column (2) of Table 4 (Individual Categories + Ability Controls). Column (1) of Table 4 is identical to the third column of Table 3 (Individual Categories).

The final specification combines all of the demographic and academic ability variables into one regression as follows:

CR - [[alpha].sub.0] + [[alpha].sub.1] (Gender, Ethnicity etc) + [[alpha].sub.2] (Academic Ability Variables) + [epsilon], (5)

and

MC = [[beta].sub.0] + [[beta].sub.1] (Gender, Ethnicity etc) + [[beta].sub.2] (Academic Ability Variables) + [epsilon]. (6)

(CR - MC) = ([[alpha].sub.0] - [[beta].sub.0]) ([[alpha].sub.1] - [[beta].sub.1])(Gender, etc) + ([[alpha].sub.2]) - [[beta].sub.2]) (Academic Ability) + [epsilon].

Column (3) of Table 4 reports coefficients for relative advantage for specification 6 (All Categories + Ability Controls). What do the results in Table 4 reveal?

Gender. As noted above, the positive coefficient for (CR - MC) in column (1) indicates a relative advantage in CR for females compared with males. This is consistent with previous studies. When controls for student ability are introduced into the separate category regression (cf. column 2), the coefficient remains positive but drops below the level of significance (p-value = 0.2309). However, when all the student categories are included in the regression (cf. column 3), the coefficient becomes significant at the 5% level again (p = 0.0125).

Ethnicity. From Table 4, all ethnic groups have a relative disadvantage compared with Europeans in CR as the coefficients are negative. However, this disadvantage disappears for all but the Asian ethnicity group when controls for student quality are introduced. When all the categories and controls for student ability are run together none of the ethnicity groups are significant (column 3 of Table 4). The fact that the coefficient for Asian becomes insignificant when all the categories are run together indicates that rather than an ethnicity issue, this is likely to be a language issue. Not all students who declare Asian as their ethnic group will be second language English speakers. Effectively, this is controlled for when all categories are run together and ethnicity becomes irrelevant.

The overall conclusion is that there is no relative disadvantage in CR for Maori, Pacific Island or Other and the use of CR questions does not discriminate against these groups.

The absolute disadvantage that non-European groups experience in both MC and CR as shown in Table 3 is therefore partly explained by a much broader struggle with University study. Juhong and Maloney (2006) find a very similar result.

So is assessment type blind to ethnicity? Clearly, ethnic groups different to the control group do not perform as well on either MC or CR but the difference is generally not significant when student ability is controlled for. This is a comforting result for instructors who wish to employ both MC and CR questions.

First language. Students with Chinese as their first language clearly have a relative disadvantage in CR. The corresponding coefficient is negative and strongly significant even after controlling for student ability. This latter result is similar to the ethnicity result, which is not surprising given that language and ethnicity are highly collinear. There are 3128 students (3) in the dataset who declare their first language to be Chinese with 3104 in the Asian ethnicity category. Hence, the Chinese category for language is almost entirely contained in the ethnicity category of Asian.

There is some weak evidence that Other language speakers also have a relative disadvantage in CR. The coefficient, when controlling for student ability, is significant at the 10% level when all categories are run together and is only just outside the same level of significance when language is run separately (p = 0.1169).

The implication of these results is that, as would be expected, non-English speaking background students do in fact face a relative disadvantage in CR questions. (4) In the case of language, this is probably reasonable. While not the direct focus of what is being assessed, CR questions will test language ability to a much greater extent than MC. The ability to interpret more open ended questions, formulate ideas and then communicate those ideas is an important skill in economics. In the end, students who graduate from the University of Canterbury do so with a degree from an English-speaking university, so being able to read, interpret and communicate in English is important. Students for whom English is not their first language and who plan to enter an English-speaking university may well find that improving their English ability prior to entry is the single most important thing they can do to improve chances of success.

International. International students appear to have a relative disadvantage in CR in the absence of the other categories. However, this is likely to be picking up the effect of language. When language, gender and ethnicity are included (specification 6) the coefficients in Table 4 become insignificant (a result that is also true even without the controls for student ability included). Being an international student per se is not likely to result in a relative disadvantage in either MC or CR.

This is not only because most of the effect is captured by language but also because not all international students have English as a second language. Of the 5902 international students, 658 declare English as their first language and 790 declare Other.

Macroeconomics and microeconomics. Table 5 shows the same results as column (3) of Table 4 but divided into microeconomics and macroeconomics. The macroeconomics results are the same as the overall results in Table 4, while the microeconomics results differ somewhat. Asian ethnicity shows an advantage in CR for microeconomics. One possible explanation is that microeconomics questions have a greater graphical, algebraic and computational element than questions in macroeconomics, which require more language skills. This may well play into the hands of students with a stronger mathematical facility, including students of Asian ethnicity. This could be consistent with the observation for females where the advantage that females have in CR disappears for microeconomics. The relative advantage for females may arise, at least in part, from their relatively better language performance. The greater premium on symbolic representation and manipulation in microeconomics would then reduce this relative advantage. (5)

4. Conclusion

This study investigates whether some types of students are unfairly disadvantaged by the question format on assessments in university economics classes. I find that in the absence of controls for student academic ability, question format appears to be biased against many types of students. However, once academic ability is accounted for, most of the estimated effects become statistically insignificant.

Where significant effects remain, these can almost always be related to English language ability. Students who do not have English as their first language have a relative disadvantage in constructed response (CR) questions compared to students with English as their first language. While course instructors are not usually aiming to test language directly, language is an important component of academic study. Students need to be able to absorb and interpret information presented in English as well as form their own ideas and express these clearly. Multiple choice (MC) questions do not test such skills to the same extent as constructed response. As such, it can be argued that the relative disadvantage experienced by non-native English speakers on CR questions reflects a valid dimension of student achievement.

The only non-language related student characteristic that remained significant after controlling for student ability was gender. I find that female students do relatively better on CR questions, but only in macroeconomics classes. This result may be of interest to instructors of principles of macroeconomics courses. The use of all-MC tests and exams will disadvantage female students in a discipline where females already perform below males.

This study raises a number of interesting questions. Why do females continue to score less than males in economics when their broader GPA is in fact higher? (6) What classroom strategies might be put in place to overcome this? What kind of language-training can help non-native English speakers to reduce the disadvantage they face on CR questions? These questions remain for future research.

DOI: 10.1080/00779954.2010.522165

Appendix 1. How the test and exam have changed over time

2002 and 2003, semester 1: Assignment 10%, term test 40%, final exam 50%. Final exam: three constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 50/100, CR worth 50/100.

2003 semester 2 and 2004: Assignment 10%, term test 40%, final exam 50%. Final exam: two constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 50/100, CR worth 50/100.

2005 and 2006: Assignment 10%, online MC 10%, term test 35%, final exam 45%. Final exam: two constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 25/75, CR worth 50/75.

2007 onwards: Assignment 10%, online MC 10%, term test 20%, final exam 60%. Final exam: two constructed response questions worth 70 in total and 30 multiple choice. Term test: 25 MC worth 25/75, CR worth 50/75.

Note that from 2007 onwards students needed to pass the final exam or get 39/80 in the test and exam combined to get a full pass. The final exam became comprehensive and included a greater coverage of term 1 material than before.

The second semester offering of microeconomics was introduced in 2003. The first semester offering of macroeconomics was introduced in 2006.

Appendix 2. Details about the construction of the demographic variables

The student management system at the University of Canterbury collects data on a range of student characteristics. Students self-report their characteristics for each year they are enrolled. Some characteristics are not compulsory to complete and so may have missing values, e.g. ethnicity. Despite this, the database provides a rich source of information with which to classify students.

A complication arises because some students take their introductory economics courses over multiple years. Reasons for this include the fact that students may choose to spread out their study, or because they fail a course. In these cases, the student management system contains multiple records, one for each year the student was enrolled in an introductory economics course. Because of the self-declared nature of the data and the fact that some fields legitimately change over time (e.g., a student may gain NZ citizenship while studying), the same student may look different from one year to the next. A 'best judgment' was used to determine the most appropriate classifications for these students. If this could not be done with reasonable certitude, the student was dropped from the sample.

Citizenship. There are three values for citizenship in the student database: (i) New Zealand, (ii) Permanent Resident, and (iii) Overseas. Permanent residents are students who do not have citizenship but are entitled to reside in New Zealand on an ongoing basis.

Ethnicity. The student database contains a primary ethnicity field that supplies the following choices: (i) European, (ii) Maori, (iii) Pacific Islander, (iv) Asian, (v) Indian, and (vi) Other. Most students self-report one of these categories. However, students have the opportunity to supply their own ethnic identification. In addition, they can select multiple ethnicities. The upshot is that the student management system assigns a value of 'Unknown' for many students' ethnicities. Nevertheless, most students were able to be assigned to an ethnicity category using other information such as citizenship type and country of citizenship.

'Indian' was eliminated as a separate ethnicity category. This category accounted for less than 50 students in the full sample. In some of the subsamples used in the empirical analyses, there were no students in this category. As a result, Indian was included in Other.

The table below summarizes how student were assigned to the ethnic categories used in this study.
Assigned ethnic
category           Ethnicity reported in student records

European           -NZ European/European/Pakeha
                   -NZ European/Pakeha
                   -Australian
                   -Unknown' and country of citizenship is any of
                     New Zealand, United Kingdom, France, United
                     States, Netherlands, Latvia, Russia, Ukraine,
                     Australia, Norway, Canada or Germany.
Asian              -Chinese
                   -Filipino
                   -Other Asian
                   -Unknown' and country of citizenship is any of
                     China, South Korea, Malaysia, Thailand, Japan,
                     Philippines, Indonesia, Pakistan, Sri Lanka
                     or Singapore.
Maori              -New Zealand Maori
Pacific Islander   -Fijian
                   -Samoan
                   -Tongan
Other              -Unknown' and country of citizenship is any of
                     Nigeria, Zimbabwe, South Africa, Kenya,
                     Tanzania, Sudan, Maldives Islands, Ethiopia,
                     Egypt, Saudi Arabia or Chile.


Language. The 'First Language' field in the student information file supplies the following categories: (i) English; (ii) Mandarin, (iii) Other Chinese Dialect, (iv) Other Asian, (v) Maori, (vi) Other, and (vii) Not Specified. As would be expected with self-reported data of this sort, the data are noisy. For example, a student from Hong Kong declared his language as Other Chinese Dialect in one year, but later identified English as his first language. Similarly, a student from Taiwan originally declared Mandarin as his first language, but reported Not Specified a later year. In many cases, these ambiguities are legitimate as many students are highly fluent in more than one language, so that there is little basis for choosing one language as 'first language' over another. Finally, Maori was included with English because there were only three students in the sample who declared Maori as their first language. All of these would be fluent in English. The table below summarizes the language categorization system used for this study.
Assigned language
category            'First language' reported in student records

Chinese             --Mandarin
                    --Other Chinese Dialect
                    --(i) Language reported as 'Not Specified',
                    'Other' or 'Other Asian;' and (ii) Citizenship =
                    'Overseas' and Country = 'China'
English             --English
                    --Maori
                    --(i) Language reported as 'Not Specified',
                    'Other' or 'Other Asian;' and (ii) Citizenship =
                    'New Zealand' and Ethnicity = 'European,' OR
                    Citizenship = 'New Zealand' and Ethnicity =
                    'Maori,' OR Citizenship = 'New Zealand' or 'United
                    Kingdom' or 'United States' or 'Canada'
Other               --(i) Language reported as 'Not Specified',
                    'Other' or 'Other Asian;' and (ii) does not meet
                    any of the conditions above

Appendix 3. Simple counts of commonly taken courses and combinations
of those courses

Course combinations                         Number   Percent

Individual Courses
Accounting                                   3226       51
Mathematics                                  2298       36
Statistics                                   4184       66
Management                                   3798       60
Combinations of two courses
Accountancy and Mathematics                  1364       22
Accountancy and Statistics                   2746       43
Accountancy and Management                   2366       37
Mathematics and Statistics                   1810       29
Mathematics and Management                   1187       19
Statistics and Management                    3008       48
Combinations of three courses
Accountancy, Mathematics and Statistics      1249       20
Accountancy, Mathematics and Management       869       14
Accountancy, Statistics and Management       2108       33
Mathematics, Statistics and Management       1078       17
All four courses                              823       13
Taken none of the four courses                792       13
Total number of individual students          6313      100

Appendix 4. Coefficients for CR and MC separately

Gender

                                  CR

                   No controls          With controls

Constant        52.3 *** (264.93)     33.37 *** (78.63)
Female          -0.84 *** (-2.81)    -2.68 *** (-12.01)
Non Econ GPA                          4.79 *** (64.38)
Accy.                                 -1.34 *** (-3.76)
Accy GPA                              0.70 *** (12.12)
Math.                                  2.63 *** (7.54)
Math GPA                                -0.03 (-0.43)
Stat.                                   -0.01 (-0.02)
Stat GPA                               0.44 *** (7.28)
Mgmt.                                 -3.44 *** (-9.52)
Mgmt GPA                              0.66 *** (10.24)
Observations          20446                 20254
[R.sup.2]             0.0004               0.4590

                                  MC

                   No controls          With controls

Constant        68.9 *** (486.37)     58.0 *** (167.37)
Female          -1.93 *** (-8.91)    -2.94 *** (-16.42)
Non Econ GPA                          2.91 *** (48.55)
Accy.                                 -1.54 *** (-5.19)
Accy GPA                               0.47 *** (9.89)
Math.                                  1.38 *** (4.76)
Math GPA                                 0.06 (1.07)
Stat.                                  1.13 *** (3.69)
Stat GPA                                -0.03 (-0.71)
Mgmt.                                -3.96 *** (-13.06)
Mgmt GPA                              0.77 *** (14.57)
Observations          20446                 20254
[R.sup.2]             0.0039               0.3376

Note: Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the 1 % level, ** at  the 5%
level and * at the 10% level.

Ethnicity

                                       CR

                       No controls           With controls

Constant            54.99 *** (275.72)     33.73 *** (78.78)
Maori Ethnicity     -5.87 *** (-6.38)      -1.58 ** (-2.25)
Asian Ethnicity     -6.28 *** (-20.40)    -3.93 *** (-14.64)
Pacific Island     -14.20 *** (-13.22)     -2.71 *** (-3.05)
Other Ethnicity     -6.53 *** (-7.94)      -2.04 *** (-3.47)
Non Econ GPA                               4.63 *** (61.40)
Accy.                                      -0.90 ** (-2.53)
Accy GPA                                   0.69 *** (11.99)
Math.                                      3.54 *** (10.20)
Math GPA                                      0.06 (0.91)
Stat.                                         0.19 (0.51)
Stat GPA                                    0.51 *** (8.47)
Mgmt.                                      -3.50 *** (-9.74)
Mgmt GPA                                    0.55 *** (8.42)
Observations              20446                  20254
[R.sup.2]                 0.0260                0.4610

                                       MC

                       No controls           With controls

Constant            70.30 *** (496.4)     57.92 *** (165.37)
Maori Ethnicity     -4.28 *** (-6.28)      -1.61 *** (-2.68)
Asian Ethnicity     -4.50 *** (-20.06)    -2.61 *** (-12.14)
Pacific Island     -10.06 *** (-11.75)     -2.92 *** (-3.87)
Other Ethnicity     -5.00 *** (-8.41)      -2.21 *** (-4.78)
Non Econ GPA                               2.80 *** (46.21)
Accy.                                      -1.31 *** (-4.42)
Accy GPA                                    0.46 *** (9.68)
Math.                                       2.10 *** (7.27)
Math GPA                                     0.09 * (1.73)
Stat.                                       1.30 *** (4.23)
Stat GPA                                      0.00 (0.04)
Mgmt.                                     -4.08 *** (-13.45)
Mgmt GPA                                   0.69 *** (12.89)
Observations              20446                  20254
[R.sup.2]                 0.0256                0.3341

Note: Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the 1% level, ** at the 5%
level and * at the 10% level.

Language

                                       CR

                    No controls           With controls

Constant            53.90 *** (295.88)     33.09 *** (78.79)
Chinese language    -5.75 *** (-17.23)    -4.54 *** (-15.42)
Other language      -5.84 *** (-10.41)     -2.15 *** (-4.89)
Non Econ GPA                               4.66 *** (62.16)
Acc                                        -0.72 ** (-2.03)
Accy. GPA                                  0.68 *** (11.77)
Math.                                      3.56 *** (10.26)
Math GPA                                     0.11 * (1.69)
Stat.                                         0.28 (0.74)
Stat GPA                                    0.50 *** (8.35)
Mgmt.                                      -3.39 *** (-9.43)
Mgmt GPA                                    0.53 *** (8.15)
Observations               20446                 20254
[R.sup.2]                 0.0169                0.4614

                                       MC

                    No controls           With controls

Constant            69.37 *** (530.07)    57.29 *** (165.87)
Chinese language    -3.62 *** (-14.77)     -2.11 *** (-8.71)
Other language       -4.06 *** (-9.72)     -1.49 *** (-4.22)
Non Econ GPA                               2.84 *** (47.10)
Acc                                        -1.34 *** (-4.49)
Accy. GPA                                   0.45 *** (9.50)
Math.                                       2.04 *** (7.02)
Math GPA                                     0.09 * (1.70)
Stat.                                       1.31 *** (4.26)
Stat GPA                                     -0.02 (-0.37)
Mgmt.                                     -4.04 *** (-13.26)
Mgmt GPA                                   0.70 *** (13.08)
Observations               20446                 20254
[R.sup.2]                 0.0134                0.3315

Note: Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the 1 % level, ** at  the 5%
level and * at the 10% level.

International versus domestic students

                                  CR

                 No controls           With controls

Constant         53.48 *** (302.83)     33.02 *** (78.62)
International     -5.50 *** (17.31)    -3.64 *** (-13.11)
Non Econ GPA                            4.70 *** (62.64)
Accy.                                   -0.96 *** (-2.69)
Accy GPA                                0.66 *** (11.59)
Math.                                    3.40 *** (9.79)
Math GPA                                   0.09 (1.38)
Stat.                                      0.20 (0.54)
Stat GPA                                 0.46 *** (7.72)
Mgmt.                                   -3.46 *** (-9.60)
Mgmt GPA                                 0.55 *** (8.50)
Observations            20446                 20254
[R.sup.2]              0.0139                0.4598

                                  MC

                 No controls           With controls

Constant         69.13 *** (545.02)    57.26 *** (165.71)
International    -3.65 *** (-15.57)     -2.02 *** (-8.97)
Non Econ GPA                            2.86 *** (47.36)
Accy.                                   -1.41 *** (-4.73)
Accy GPA                                 0.45 *** (9.37)
Math.                                    2.00 *** (6.91)
Math GPA                                  0.09 * (1.81)
Stat.                                    1.29 *** (4.20)
Stat GPA                                  -0.03 (-0.68)
Mgmt.                                  -4.05 *** (-13.31)
Mgmt GPA                                0.70 *** (13.15)
Observations            20446                 20254
[R.sup.2]              0.0116                0.3315

Note: Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the 1 % level, ** at the 5%
level and * at the 10% level.

All demographic variables

                                       CR

                        No controls          With controls

Constant             55.1 *** (239.17)     34.33 *** (78.88)
Female                 -0.07 (-0.25)       -2.14 *** (-9.47)
Maori Ethnicity      -5.93 *** (-6.44)      -1.48 ** (-2.10)
Asian Ethnicity      -3.47 *** (-6.06)     -1.67 *** (-3.86)
Pacific Island      -13.55 *** (-12.48)     -2.13 ** (-2.41)
Other Ethnicity      -5.24 *** (-6.07)      -1.38 ** (-2.25)
Chinese language     -2.02 *** (-3.08)     -2.10 *** (-4.23)
Other language       -2.43 *** (-3.52)       -0.52 (-0.98)
International        -1.68 *** (-3.19)     -1.07 *** (-2.74)
Non Econ GPA                                4.63 *** (61.54)
Accy.                                       -0.61 * (-1.71)
Accy GPA                                    0.69 *** (11.98)
Math.                                       3.29 *** (9.41)
Math GPA                                     0.15 ** (2.32)
Stat.                                         0.19 (0.50)
Stat GPA                                    0.52 *** (8.76)
Mgmt.                                      -3.31 *** (-9.24)
Mgmt GPA                                    0.53 *** (8.18)
Observations               20446                 20254
[R.sup.2]                  0.0280                0.4650

                                       MC

                        No controls          With controls

Constant            70.90 *** (437.90)    58.80 *** (166.71)
Female               -1.47 *** (-6.76)    -2.70 *** (-14.85)
Maori Ethnicity      -4.24 *** (-6.21)     -1.45 ** (-2.40)
Asian Ethnicity      -3.59 *** (-8.47)     -2.21 *** (-6.34)
Pacific Island      -9.63 *** (-11.20)     -2.46 *** (-3.26)
Other Ethnicity      -4.34 *** (-6.95)     -1.97 *** (-4.10)
Chinese language        0.16 (0.33)           0.56 (1.40)
Other language        -0.90 * (-1.76)         0.36 (0.86)
International        -1.11 *** (-2.88)     -0.63 ** (-1.98)
Non Econ GPA                               -2.81 *** (46.62)
Accy.                                      -1.22 *** (-4.11)
Accy GPA                                    0.47 *** (9.91)
Math.                                       1.69 *** (5.82)
Math GPA                                    0.13 ** (2.47)
Stat.                                       1.20 *** (3.93)
Stat GPA                                      0.01 (0.27)
Mgmt.                                     -3.94 *** (-13.15)
Mgmt GPA                                   0.71 *** (13.31)
Observations               20446                 20254
[R.sup.2]                 0.0285                0.3415

Note: Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the 1 % level, ** at  the 5%
level and * at the 10% level.


Acknowledgements

The author wishes to acknowledge Bob Reed for his support and extremely helpful comments with this paper.

References

Anderson, G., Benjamin, D., & Fuss, M.A. (1994). The determinants of success in university introductory economics courses. Journal of Economic Education, 25(2), 99 119.

Juhong, B., & Maloney, T. (2006). Ethnicity and academic success at university. New Zealand Economics Papers, 40(2), 181-213.

Lumsden, K.G., & Scott, A. (1987). The economics student reexamined: Male-female differences in comprehension. Journal of Economic Education, 18(4), 365 375.

Walstad, W.B., & Robson, D. (1997). Differential item functioning and male-female differences on multiple-choice tests in economics. Journal of Economic Education, 28(2), 155 171.

Ziegert, A.L. (2000). The role of personality temperament and student learning in principles of economics: Further evidence. Journal of Economic Education, 3l(4), 307-322.

Notes

(1.) Broadly speaking, school decile is a measure of the overall affluence of the school that the student attends. The New Zealand Ministry of Education website (www.minedu.govt.nz) has the following statement:

'A school's Decile indicates the extent to which it draws its students from low socioeconomic communities. Decile 1 schools are the 10% of schools with the highest proportion of students from low socio-economic communities. Decile 10 schools are the 10% of schools with the lowest proportion of these students.'

(2.) One possible difficulty with using a variable like NEGPA is that this may, like the dependent variable under study, be prone to error due to the omission of an underlying and probably unobservable ability variable, which means that the error term is likely to be correlated with the NEGPA variable. This brings the usual problems for the OLS estimates. However, the primary effect of unobservable ability should be minimized via the differencing of CR and MC scores. If I rewrite equations (1a) as follows:

[CR.sub.i] = [[alpha].sub.0] + [[alpha].sub.1] [Gender.sub.i] [[lambda].sub.i] + v

[MC.sub.i] = [[beta].sub.0] + [[beta].sub.1] [Gender.sub.i] + [[lambda].sub.i] + [mu]

where [lambda]i is the unobservable ability variable. Then

([CR.sub.i] - [MC.sub.i]) = ([[alpha].sub.0] - [[beta].sub.0]) + ([[alpha].sub.1] -/ [[beta].sub.1]) [Gender.sub.i] + (v - [mu]).

The preceding assumes that ability has the same effect on CR and MC scores. If, instead, ability has a differential effect on the two scores, then the corresponding difference equation is

([CR.sub.i] [MC.sub.i]) - ([[alpha].sub.0] - [[beta].sub.0]) + ([[alpha].sub.1] - [[beta].sub.1]) [Gender.sub.i] + ([[lambda].sub.i,CR] - [[lambda].sub.i,MC]) [Ability.sub.i] + (v - [mu]).

If my ability controls do not control for all relevant ability influences, and these unobserved ability influences are correlated with my controls (such as NEGPA), then it is correct that the error term will likely be correlated with the included ability controls. This will generate biased estimates of the ability control coefficients. However, as long as these unobservable ability influences are uncorrelated with the gender/ethnicity/etc, variables, these latter coefficients--which are the main objects of my analysis should be unbiased.

(3.) This number is calculated by treating a student who appears in multiple years as a different student in each year since self-declared demographic characteristics can change from year to year.

(4.) While this result holds over all the data it may not hold at every point in time or in every particular course offering. The main exception is the 2004 year where the relative disadvantage is in MC rather than CR for Chinese language speakers. In 2003, the percentage of Chinese language students climbed to 40% of the Microeconomics class as the number of Chinese students entering NZ education rose dramatically. It is possible that, consciously or otherwise, an over-compensation was made by the instructor in the following year in response to this increase in either question setting or marking. Another possibility is that the students who arrived in this 'bubble' were qualitatively different to the long-term average Chinese speaking student. The exact cause is difficult to determine and it may simply be 'noise' in the data.

(5.) Confounding this somewhat is that females in the sample who have taken the common first year mathematics paper have a mean GPA of 4.4 compared with the mean for males of 3.5 in the mathematics course. Nevertheless, it can still be the case that the lower emphasis on language skills by virtue of the presence of an emphasis on mathematical-type skills is sufficient to remove the advantage in CR.

(6.) The mean value for NEGPA for females is 3.8 (s.e. = 0.038) but it is only 3.5 (s.e. = 0.033) for males. GPA is measured on a scale of-I to 9 where a grade of E has a GPA value of 1, D is 0, C- is 1 and so on up to an A+ which has a GPA value of 9.

Stephen Hickson *

Department of Economics and Finance, University of Canterbury, Private Bag 4800, Christchurch 8042, New Zealand

(Received 23 March 2010; final version received 28 July 2010)

* Email: stephen.hickson@canterbury.ac.nz
Table 1. How New Zealand universities assess principles of
economics courses. The following information on invigilated
assessment was obtained via an email survey. It was  correct at
December 2009 but may have subsequently changed.

                                                    Term test   Final
University     Paper                               (S) (% MC)    exam

Auckland (1)   Microeconomics                           0         0
               Macroeconomics                           0         0
               General Economics                       60        30
               BBIM General Economics                   0         0
AUT (2)        Economics                                0
Waikato        Business Economics and the NZ           60        60
                 Economy
               Economics and Society                    0         0
Massey         Microeconomics                           0       100
               Macroeconomics                           0       100
Victoria       Microeconomics                         100        70
               Macroeconomics                         100        70
Lincoln        Introduction to Applied Economics    40/50        30
               Introduction to Economic Theory      50/65        30
Canterbury     Microeconomics                          33        30
               Macroeconomics                          33        30
Otago          Principles of Economics 1               60        67
               Principles of Economics 11              50        50

Notes:

(1) University of Auckland

The two general economics papers are not taken by economics
majors. Economics majors take ECON 101 Microeconomics and ECON 111
Macroeconomics. BBIM stands for Bachelor of Business
Information Management.

(2) Auckland University of Technology

AUT operate a different teaching model with smaller classes. They
have four assessments within the economics section, namely: an
online assignment worth 10%; an invigilated test worth 20% (all
CR); a portfolio worth 30%; and an essay worth 40%.

Table 2. Summary statistics by student category.

Category          Percent of     Term test    Final exam
                    sample      CR percent     CR percent
Gender
Female               43.3          48.8           54.0
Male                 56.7          50.1           54.4
Ethnicity
Asian                39.1          46.5           50.9
European             52.8          52.5           57.4
Maori                 2.5          46.8           51.4
Pacific Island        1.6          38.4           43.1
Other                 4.0          45.6           51.3
First Language
Chinese              26.7          45.8           50.5
English              65.3          51.6           56.2
Other                 8.0          45.4           50.8
International
Domestic             71.1          51.1           55.9
International        28.9          45.8           50.2

Category           Term test    Final exam
                  MC percent     MC percent
Gender
Female               65.7           68.2
Male                 67.6           70.2
Ethnicity
Asian                65.1           66.5
European             68.6           72.0
Maori                64.0           68.1
Pacific Island       58.8           61.7
Other                63.3           67.3
First Language
Chinese              65.0           66.5
English              67.8           70.9
Other                64.6           66.1
International
Domestic             67.5           70.7
International        64.9           66.1

Note: Term Test and Final Exam Marks range from 0 to 100.

Table 3. Estimation of coefficients for CR and MC and relative
advantage (CR--MC) for separate demographic variables.

                     (1. CR) Coefficients       (1. MC) Less
                            for CR          coefficients for MC
                       ([[alpha].sub.1])      ([[beta].sub.1])
Gender (a)
Female                 -0.84 *** (-2.81)      -1.93 *** (-8.91)
Ethnicity (b)
Maori                  -5.87 *** (-6.38)      -4.28 *** (-6.28)
Asian                 -6.28 *** (-20.40)     -4.50 *** (-20.06)
Pacific Island        -14.20 *** (-13.22)    -10.06 *** (-11.75)
Other                  -6.53 *** (-7.94)      -5.00 *** (-8.41)
First Language (c)
Chinese               -5.75 *** (-17.23)     -3.62 *** (-14.77)
Other                 -5.84 *** (-10.41)      -4.06 *** (-9.72)
International (d)
International         -5.50 *** (-17.31)     -3.65 *** (-15.57)
Observations                20,446                 20,446

                         (2) Equals relative
                     advantage ([[alpha].sub.1]-
                           [[beta].sub.1])
Gender (a)
Female                      1.09 *** (4.68)
Ethnicity (b)
Maori                      -1.59 ** (-2.13)
Asian                     -1.78 *** (-7.24)
Pacific Island            -4.14 *** (-4.86)
Other                     -1.54 *** (-2.61)
First Language (c)
Chinese                   -2.13 *** (-7.88)
Other                     -1.78 *** (-4.06)
International (d)
International             -1.85 *** (-7.12)
Observations                    20,446

Notes: (1) The three coefficients in Panel a (Gender) correspond
to [[alpha].sub.1], [[beta].sub.1], and
([[alpha].sub.1]-[[beta].sub.1]) in specifications (1a) and (2a)
respectively. Panels b-d are interpreted similarly.

(2) Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the l % level, ** at the 5%
level and * at the 10% level.

Table 4. Estimation of the relative advantage associated with CR
questions for specific student groups.

                          Individual       Individual categories +
                        categories (1)      Ability controls (2)
Gender (a)
Female                 1.09 *** (4.68)           0.27 (1.20)
Ethnicity (b)
Maori                  -1.59 ** (-2.13)          0.03 (0.04)
Asian                 -1.78 *** (-7.24)       -1.32 *** (-5.02)
Pacific Island        -4.14 *** (-4.86)          0.21 (0.25)
Other                 -1.54 *** (-2.61)          0.17 (0.31)
First Language (c)
Chinese               -2.13 *** (-7.88)       -2.43 *** (-8.11)
Other                 -1.78 *** (-4.06)         -0.65 (-1.54)
International (d)
International         -1.85 *** (-7.12)       -1.62 *** (-5.78)
Observations                20,446                 20,254

                        All categories +
                      Ability controls (3)
Gender (a)
Female                  0.57 *** (2.50)
Ethnicity (b)
Maori                    -0.03 (-0.05)
Asian                     0.54 (1.33)
Pacific Island            0.33 (0.39)
Other                     0.59 (1.02)
First Language (c)
Chinese                -2.66 *** (-5.54)
Other                   -0.88 * (-1.75)
International (d)
International            -0.44 (-1.15)
Observations                 20,254

Notes: (1) Column (1) reports the estimated values of
([[alpha].sub.1]-[[beta].sub.1]) in specifications (2a)-(2d) in
the text. Column (2) reports the same after controlling for
Ability variables (see specifications (4a)-(4d) in the text).
Column (3) reports estimates of (([[alpha].sub.1]-[[beta].sub.1])
corresponding to specification (6) in the text.

(2) Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the 1 % level, ** at the 5%
level and * at the 10% level. See the text for the description of
ability control variables.

Table 5. Estimation of the relative advantage associated with CR
questions for specific student groups: macro and micro.

                   All Categories + Ability Controls

                   Macroeconomics      Microeconomics

Female              1.16 *** (3.44)       0.30 (0.99)
Maori Ethnicity      -0.49 (-0.45)        0.24 (0.26)
Asian Ethnicity      -0.92 (-1.49)      1.65 *** (3.03)
Pacific Island       -1.75 (-1.25)       1.83 * (1.70)
Other Ethnicity      -0.49 (-0.56)       1.38 * (1.79)
Chinese language   -1.64 ** (-2.26)    -3.39 *** (-5.29)
Other language        0.61 (0.82)      -1.97 *** (-2.91)
International         0.34 (0.59)       -1.11 ** (-2.14)
Observations             8904                11350

Notes: (1) Estimates in each column correspond to
([[alpha].sub.1]-[[beta].sub.1]) from specification (6) in the
text. (2) Values in parentheses are t-statistics calculated using
heteroscedastic-robust (White) standard errors. *** indicates
significant difference from zero at the 1 % level, ** at the 5%
level and * at the 10% level. See the text for the description of
ability control variables.
COPYRIGHT 2010 New Zealand Association of Economists
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2010 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:RESEARCH ARTICLE
Author:Hickson, Stephen
Publication:New Zealand Economic Papers
Article Type:Report
Geographic Code:8NEWZ
Date:Dec 1, 2010
Words:7335
Previous Article:Choice experiment assessment of public expenditure preferences.
Next Article:The spirit level: Why more equal societies almost always do better.
Topics:

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters