An incentives-based approach to implementing Financial Fitness for Life in the Milwaukee public schools.

Abstract

This paper reports on the research aimed at measuring the effect of financial incentives provided to teachers in implementing a financial and economic education curriculum in a large urban school district. The initiative involved paying teachers to attend a two-day training program and to allow their students to be pre- and post-tested using a national, standardized test. The testing was conducted to provide empirical evidence regarding the implementation of the curriculum. The statistical analysis of the pre-and post-test scores revealed that students' knowledge gains were statistically significant and that they significantly outperformed students who did not participate in the program.

JEL Codes: A2, A29, D73

Keywords: Economic education, Financial education, Incentives

I. Introduction

The proposition that people respond to incentives in predictable ways is not very controversial among most economists. In public education, however, the role of incentives has been hotly debated.

This paper reports on research aimed at measuring the effect of incentives in implementing a financial and economic education curriculum published by the National Council on Economic Education called Financial Fitness for Life (FFL) (Flowers & Szot Gallaher, 2001) in the Milwaukee Public Schools (MPS). The MPS had adopted this curriculum for use in its seventh grade social studies program in 2004, and although the materials were purchased and teachers were trained, very few teachers actually implemented the program. With help from the Northwestern Mutual Foundation, it was decided that, with most principals and grade seven teachers seemingly ignoring central office policy, perhaps a voluntary, incentives-based approach would work better. The new initiative involved paying MPS teachers to attend a two-day training program and to allow us to pre-and post-test their students using a national, standardized test of financial and economic understanding developed for this curriculum. The testing was conducted to provide empirical evidence regarding the implementation of the curriculum. The statistical analysis of the pre-and post-test scores revealed that students' knowledge gains were statistically significant and that they significantly outperformed students who did not participate in the program.

Thus, the primary research question in this study was: Would the addition of incentives in the form of bonus payments to teachers result in the voluntary implementation of an economic and personal finance curriculum in an urban school district in which previous efforts had failed?

II. Related Research

What is the role of incentives in education? Do different forms of compensation matter? The landmark Coleman Report (Coleman, Campbell, Hobson, McPartland, Mood, Weinfield and York, 1966) implied that school governance matters such as incentives and teacher effort may have little effect on academic achievement. It concluded that students' family backgrounds trumped all other variables in terms of school outcomes. One inference was that poverty and ethnicity exerted a powerful effect on academic achievement. Family background and neighborhood environment mattered more than school governance or teacher effort.

The Coleman Report (Coleman, Campbell, Hobson, McPartland, Mood, Weinfield and York, 1966) touched off a debate. Educational researchers and economists disagreed on the role of incentives in education. Studies of teacher incentives appear to fall into two groups. The first group includes advocates for alternative forms of teacher compensation to reward performance. This group offers little empirical support for its claims (Odden and Kelly, 2002). The second group seeks to verify or refute the research of Eric Hanishek's controversial finding that "money doesn't matter" in evaluating the impact of teacher compensation on student performance (Hanushek, Kain, and Rivkin, 1999). Research in this second group is most concerned with correcting misspecifications in education production function models but analyzes only traditional teacher compensation plans.

Research at the University of Arkansas' Department of Education Reform (Winters, Ritter, Barnett, and Green, 2007) recently weighed in favoring the first group. A teacher bonus program in the Little Rock, Arkansas, public schools succeeded in linking teacher merit pay to student test scores. In this program, a 4 percent improvement in achievement scores earned a $100 bonus per student, rising to $400 if the student gained 15 percent. The authors found that providing teachers with bonuses based on test score improvements significantly increased student math proficiency in comparison to the performance of students attending similar schools that were not offered the chance to participate in the program.

In the current study, we wanted to see if the addition of teacher incentives would result in the classroom implementation of the FFL economics and financial education curriculum and result in measurable knowledge gains for students.

III. An Incentives-Based Approach

Could the addition of incentives in the form of bonus payments to teachers result in the voluntary implementation of an economic and personal finance curriculum in an urban school district in which previous efforts had failed? We decided to give it a try. In year one of the program (2005-2006), we recruited ten teachers from five MPS middle schools, and nearly 600 students participated. The teachers were offered a financial incentive to participate. The University of Wisconsin-Milwaukee Center for Economic Education conducted the training program for the teachers in the fall of 2005. Teachers were trained to use a curriculum published by the National Council on Economic Education called Financial Fitness for Life: Shaping Up Your Financial Future (FFL) for grades 6-8. FFL includes 17 lessons divided into five theme areas: the economic way of thinking, earning an income, saving, spending and using credit, and managing money. These lessons were reported to be carefully designed and field tested so as to be age-appropriate. The training stressed basic concepts of economics and personal finance, provided a detailed overview of the FFL lessons, and explained the pre- and post-testing procedures. Afterward, one of the researchers visited each class to administer the pre- and post-tests. The teachers never saw the test. Teachers had been cautioned during the training that they were not to inquire about the test itself.

In year two (2006-2007), the program was increased to include 22 teachers and more than 1,000 students. Six of the teachers from year one continued their involvement in year two. Teachers were again offered a financial incentive to participate in a two-day training program and to allow the pre- and post-testing of their students.

IV. Test Instrument

FFL is accompanied by a 50-item, standardized, multiple choice test published by the National Council on Economic Education (Walstad and Rebeck, 2005). This test was carefully designed and developed to cover the subject matter that should be taught when using the FFL curriculum. Substantial evidence is available regarding the reliability and validity of the test. The National Advisory Committee that developed the test items took several steps to establish content, construct, and criterion-related validity.

In the two years of the program, the pre-tests were administered in October, and the post-tests were administered in December.

V. Design

Seventh grade teachers were recruited from MPS to participate. MPS is a large urban district of 85,000 students. It faces many of the same problems as do other such districts, including a low high school graduation rate and poor student performance on state tests in reading and mathematics.

School-based Learning Coordinators were invited to nominate grade seven teachers to participate in the program. Subsequently, several telephone calls and e-mails were made to finally identify the teachers who were wished to be involved. Teachers were asked to teach the FFL curriculum to their students. These students became members of the treatment groups. Other teachers within the same schools agreed not to teach to FFL curriculum to all their students at this time. These students were pre- and post-tested but were not taught the FFL curriculum. These students became members of the control group.

VI. Results

Year one of the program involved 10 teachers with 624 students in the treatment group and 94 in the control group. The results of year one were generally positive. Here is a brief overview. The aggregate mean score of the FFL student group (m = 18.07, n= 572) and aggregate mean score of the students in the control group (M = 19.42, n=89) was tested using paired-samples t-test. This analysis was applied only to the tests of those students who had completed both the pre-test and the post-test. The Paired-Samples T-Test measures the mean "gain" score between the pre-test and post-test results for each student. Testing the aggregate FFL group, the difference between the pre-test and post-test scores was found to be significant, t (571) = 21.638, p <0.001. Conversely, the results for the control group by Paired Samples T-Test, the difference between the pre-test and post-test scores, was found to be non-significant, t (88) = 1.033, p = 0.304. So, it appeared from our initial results that the FFL curriculum had, in fact, been implemented in classrooms and had a positive influence on student knowledge.

The second year of the program involved 22 teachers with 1,031 students in the treatment group and 104 in the control group. Here is a more detailed report on the second year of the program. Several statistical procedures were used to measure the pre- and post-test scores. We compared the pre-and post-test mean scores of all students who participated in the FFL curriculum to all the students in the control group. Table 1 shows that the treatment group (all students that had received the FFL curriculum) had a mean pre-test score of 18.01 and a mean post-test score of 22.14, with a mean gain of 4.12 ( n = 1031, SD = 5.983 ). The control group had a mean pretest score of 19.19 and a mean post- test score of 18.86, with a mean gain of -0.23 (n = 104, SD = 4.316). This was a positive outcome, but more tests were necessary before we could conclude that this change was statistically significant and had not occurred by chance.

Prior to conducting more elaborate statistical tests on the mean scores, we wanted to check if the data were distributed in a normal fashion. This step was necessary to help us decide what additional statistical tests were appropriate. This check for normality was done in two ways. First, the data were examined visually by q-q plot. This test suggested that the data were distributed in a normal fashion. Second, a test called the Kolmogorov-Smirnov test of normality was used to examine the normality of the aggregate treatment group and the individual teacher groups. We again determined the data to be normally distributed. As a result, it was appropriate for us to use the t-test to measure the effects of the FFL curriculum on student test scores. A t-test is a commonly used statistic that is used to check for statistically significant changes in mean scores.

The aggregate treatment group (n=1,031) and the control group (n=104) scores were tested using a Paired Samples T-Test. This analysis utilizes only the scores of those students who had completed both a pre- and post-test. The Paired Samples T-Test uses the mean "gain" score between the pre-test and post-test and compares it to zero. The difference between the pre-test and post-test scores of the aggregate treatment group was found to be significant, t (1030) = 22.131, p <0.001. Conversely, the results for the control group by Paired Samples T-Test were found to be non-significant, t (103) = 0.545, p = 0.587. It appeared from these additional results that the FFL curriculum had, in fact, been implemented in classrooms and had a positive influence on student knowledge.

Finally, we examined the mean scores on the pre-and post-tests by each teacher in the treatment groups again using the Paired Samples T-Test. Table 2 shows that 13 out of 19 classrooms had statistically significant gain scores. The classes that did not show statistically significant gains had two special conditions. In one case, there were larger than average numbers of special education (Learning Disabled and Learning Disability) students. In a second case, the teachers had been reassigned to teach other subjects, and we questioned the extent of their use of the FFL curriculum.

VII. Limitations

Of course, a study of this sort has several limitations. The teachers were not randomly selected or randomly assigned to their classes. Also, students were not randomly assigned to their teachers or to the treatment or the control groups. A better experiment to evaluate the value of the FFL curriculum would have included a treatment group that taught economics and personal finance without using the FFL materials. However, the main purpose of this study was not to measure the effectiveness of the curriculum but, instead, to see if participation in an incentive-based program would result in the implementation of an economics and personal finance curriculum.

VIII. Conclusions

Despite these limitations, we think that some important results can be inferred from this experience. Students who participated in the FFL curriculum appear to have significantly improved their knowledge of economics and personal finance when compared to students who were in the classes of teachers who did not use the FFL materials. These results lead to two important conclusions. First, an incentive-based approach seems to be an effective way to gain the cooperation of teachers in implementing a new economic and financial education curriculum. This is not a trivial result. Examples of failed efforts to implement a new curriculum in large urban school districts are widespread. Second, it is clear that urban youth can learn key concepts and principles of economic and financial education if instruction is provided using appropriate curriculum materials.

This is good news to people who think that they can do little to improve the economic prospects of students in large urban school districts. People who consider the current state of economic and financial literacy to be unacceptably low may find that an incentives-based approach is a great way to get started.

References

Coleman, James. S., Ernest Q. Campbell, Carol F. Hobson, James M. McPartland, Alexander M. Mood, Frederic D. Weinfield, and Robert L. York. 1966. Equality of Educational Opportunity. Washington D.C.: U.S. Printing Office.

Flowers, Barbara, and Sheryl Szot Gallaher. 2001. Financial Fitness for Life: Shaping Up Your Financial Future. New York: National Council on Economic Education.

Hanushek, Eric, John F. Kain, and Steven G. Rivkin. 1999. "Do Higher Salaries Buy Better Teachers?" Paper presented at the Annual Meeting of the American Economic Association, New York.

Odden, Allan, and Carolyn Kelly. 2002. Paying Teachers for What They Know and Do: New and Smarter Compensation Strategies to Improve Schools. Thousand Oaks, CA: Corwin Press, Inc.

Walstad, William, B. and Ken Rebeck. 2005. Financial Fitness for Life: Middle School Text Examiner's Manual. New York: National Council on Economic Education.

Winters, Marcus A., Gary W. Ritter, Joshua H. Barnett, and Jay P. Green. 2007. An Evaluation of Teacher Performance Pay in Arkansas. University of Arkansas. www.uark.edu/ua/der/Research/performance_pay_ar.html (accessed February 15, 2008).

Nicole M. Butt

University of Wisconsin-Milwaukee

Stephen J. Haessler

The Robert and Marie Hansen Foundation

Mark C. Schug

University of Wisconsin-Milwaukee

Table 1: Mean Pretest and Posttest Scores for Test Groups

                     Pretest               Posttest

Test Group       M      SD      n       M      SD      n

Aggregate
  Control      19.19   5.632    120   18.86   5.858    115
Aggregate
  Treatment    18.01   6.008   1243   22.14   7.848   1120
Teacher A      17.81   4.926     89   18.71   5.957     75
Teacher B      19.49   5.939     93   21.91   6.708     76
Teacher C      16.85   5.070     34   17.82   6.262     33
Teacher D      14.78   4.081     18   15.57   6.047     14
Teacher E      19.61   5.579    103   26.37   7.470    106
Teacher F      18.23   4.722     35   17.21   3.658     29
Teacher G      16.57   5.783     53   19.41   6.830     46
Teacher H      17.90   5.497     79   20.01   7.587     73
Teacher I      19.93   4.373     41   27.93   6.698     42
Teacher J      15.29   5.183     38   18.00   6.124     17
Teacher K      17.77   4.964     57   25.96   7.936     50
Teacher L      15.48   5.178     73   20.05   8.332     74
Teacher M      16.29   5.028     83   23.03   7.332     74
Teacher N      17.18   5.081     61   25.36   8.145     59
Teacher 0      15.66   5.043     74   17.91   5.564     69
Teacher P      14.21   4.611     76   17.93   5.977     60
Teacher Q      13.14   4.597     21   14.00   3.952     22
Teacher R      24.63   6.987    101   26.13   7.404     96
Teacher S      19.70   5.993    115   25.48   7.162    108

* note: the variation in the number of test subjects from pretest
to posttest is due to the variation in attendance of students

Table 2: Paired Samples T-test Results by Teacher Group

             Mean Gain     t      df=    SD      p=
               score

Teacher A      1.145      1.850    68   5.140   0.069
Teacher B      2.141      4.018    70   4.489   0.000
Teacher C      0.200      0.166    29   6.609   0.870
Teacher D      0.571      0.348    13   6.136   0.733
Teacher E      6.667     12.534   101   5.372   0.000
Teacher F     -1.448     -1.913    28   4.076   0.066
Teacher G      3.750      5.741    39   4.131   0.000
Teacher H      1.889      2.755    62   5.442   0.008
Teacher I      8.108      8.365    36   5.896   0.000
Teacher j      1.941      1.938    16   4.130   0.070
Teacher K      8.065      9.276    45   5.897   0.000
Teacher L      5.446      6.662    64   6.591   0.000
Teacher M      6.773     10.011    65   5.496   0.000
Teacher N      8.035      9.168    56   6.617   0.000
Teacher 0      2.607      3.533    55   5.522   0.001
Teacher P      3.393      4.499    55   5.643   0.000
Teacher Q      0.667      0.597    17   4.740   0.559
Teacher R      1.892      3.852    92   4.738   0.000
Teacher S      5.942     12.235   102   4.929   0.000