Constructing analysis of variance (ANOVA).

Spreadsheets can be a valuable tool for helping students construct a deeper understanding of statistical concepts. This investigation incorporated active-learning into the structure of an intermediate-level statistics course for doctoral students in education. Active-learning was implemented primarily by having students create learning playgrounds, which would instruct a novice about specific statistical concepts. This format led these doctoral-level education participants to become more fully involved in the process of mathematical storytelling with the beneficial consequence of a richer understanding of key analysis of variance concepts and techniques.

**********

One of the biggest problems in statistics education is that students tend to suffer from inert knowledge. Whitehead (1929) first used this term to describe knowledge, which can usually be recalled by students when explicitly asked to do so, but it is not used spontaneously in problem solving even though the knowledge is relevant. A new graduate student in the social sciences may take only two classes in statistical analysis and research design. A year or so later the student will then be expected to make use of that knowledge, either as a research assistant on a project or in their own research study. All too often students respond to these applied situations as if they had never taken a statistics or research design course. Although they likely remember key statistical concepts, the process of taking data and transforming it into meaningful information is frustrating and frightening to many. Of course this description is a general "best case" scenario. Using statistics goes beyond simply remembering knowledg e. Some of the ideas in statistics, and especially their connection to research design, are often never well understood by students.

In addressing the general problem of inert knowledge, Dewey made the case for an active-learning approach in which students are learning new material for some practical project or need in the present. As Dewey put it:

When preparation is made the controlling end, then the potentialities of the present are sacrificed to a suppositious future. When this happens, the actual preparation for the future is missed or distorted. The ideal of using the present simply to get ready for the future contradicts itself. It omits, and even shuts out, the very conditions by which a person can be prepared for his future. We always live at the time we live and not at some other time, and only by extracting at each present time the full meaning of each present experience are we prepared for doing the same thing in the future. This is the only preparation which in the long run amounts to anything. (1963, p. 49)

Benware and Deci (1984) hypothesized that active-learning where the student is "...learning material to teach it will lead to enhanced learning and to a more positive emotional tone than learning material to be tested on it, even when the amount of exposure to the material being learned is the same" (p. 756). Benware and Deci built their study upon the theoretical approaches of Bruner (1966) and Rogers (1969) who both suggested that students learn better if the content of the instruction is useful for a task they are undertaking. The "activity" would in turn result in a fuller engagement of the material. The logic behind this line of thinking is fairly simple: students approach the material with the anticipation of using it, so they become more fully involved.

This article presents an investigation into an active-learning approach to statistics through the use of spreadsheet software. The focus of the intermediate-level statistics course used in this study was on learning analysis of variance (ANOVA) techniques. In the past this course had been taught through a combination of lecture, computer-based activities with SPSS, and structured activities with Excel. The instructor (author) hypothesized that increasing the level of active-learning in the course through problem-based challenges would result in greater learning and higher student motivation. Indeed, previous research has indicated that active-learning approaches can be quite effective (Benware & Deci, 1984; Brophy & Alleman, 1991; Kafai, 1995; Mitchell, 1993; Mitchell, 1997).

Excel is a powerful spreadsheet software program that allows individuals to do more than simply crunch numbers. Through Excel's open-endedness, the ability to incorporate conceptual formulas, the use of intuitive naming of cells and arrays, dual coding features, and Excel's design tools, students can potentially create rich educational products. This full-featured software program appeared to offer a great way to pragmatically implement an active-learning curriculum. Specifically, students actively learned statistical concepts by creating learning playgrounds that would instruct a novice about particular statistical techniques. This kind of learning challenge was hypothesized to result in greater learning for the "constructors," or students, in the course.

In the present investigation, active-learning was used primarily for out-of-classroom activities that students completed. The classroom experience itself used instructor-led multimedia presentations, discussions, and computer-based examples. Indeed, the in-class experience could be described as being teacher-led because there was relatively little opportunity for true active-learning to take place. Instead it was the way students spent engaging with the course material outside of the classroom that incorporated the active-learning challenges.

Two studies previous to Benware and Deci tested the active-learning hypothesis and found positive indications that active-learning is effective (Bargh & Schul, 1980; Zajonc, 1960). However both of those studies used very short treatment periods. Benware and Deci's study represented the first systematic attempt to test the "learning-to-teach" hypothesis using a reasonable treatment period of three hours. Their results confirmed that students under the experimental "active" condition learned both rote and conceptual material significantly better than the control group. Just as importantly, the Benware and Deci study incorporated motivational variables and found that students in the experimental condition found the process of learning more interesting and enjoyable than those in the control group.

More recently, Mitchell (1997) described a classroom learning environment in which students learned about a computer-based approach to learning statistics through Microsoft Excel. The students in that study created educational worksheets. Mitchell highlighted five factors that helped explain the advantages of spreadsheet software in creating an active-learning based environment including creating multiple representations of statistical measures, making "number playgrounds," incorporating story lines, and the opportunity for student creativity. The result of making such products resulted in a type of active-learning in which students' thinking about basic statistical concepts grew much richer through the development of these products.

The present study sought to build upon the Benware and Deci (1984) and Mitchell (1997) studies with three important enhancements. First, like Mitchell, the study covers a time period of one semester as opposed to the three hour treatment used by Benware and Deci. Second, like Mitchell, the study used statistics as the content area, a subject that many students find difficult to understand and motivationally unappealing. However, unlike Mitchell, this study takes place in the context of a regular course focusing on statistical concepts rather than using a course that is primarily computerbased in its content. Thus this study hoped to address the more realistic concern of integrating technology into regular content courses. Finally the focus of this study is on a content analysis of students' products: did these products demonstrate significant understanding of the course concepts?

The challenge faced by intermediate-level students in this study was to create learning playgrounds that would instruct novice doctoral students about analysis of variance techniques. Students were told by the instructor that the best products resulting from the class would be used as learning aids in his beginning level statistics courses in the future. Since all the students were involved in education at the K-12 or higher education levels, there was the additional incentive for them to take advantage of their educational expertise to create a compelling set of learning playgrounds. This active-learning project took place within the context of using spreadsheet software as their instructional creation tool. The benefits of using spreadsheets, specifically Excel, to accomplish these goals was crucial.

THE POWER OF SPREADSHEETS

In this course spreadsheets were used as a pedagogical tool. For the purposes of statistical analysis only a software program such as SPSS or SAS would be much better choice. Excel is more akin to a numbers-based LOGO program. Just as Kafai (1995) and Papert (1980) had taken advantage of the relatively simple programming language called LOGO to teach elementary level kids about mathematics, so in this study Excel was used to help students learn about statistical concepts in a hands-on manner. Kafai's study was particularly relevant because she spent six months helping a fourth-grade class learn how to use LOGO. Those students had a special challenge: to create a learning environment that would teach younger students at their school about fractions. Kafai found significant results in both learning and motivation when students were faced with such a concrete and worthy challenge. Many students went to painstaking lengths to create LOGO products that were "special." The final products created by these students w ere truly impressive.

A number of researchers have used spreadsheets as tools for exploring mathematical concepts. For instance, Dugdale (1998) discussed a class in which spreadsheets proved to be a highly effective tool for middle-grade students learning about sequences and series. Similarly Abramovich (1995), Abramovich & Nabors (1998), and Sutherland & Rojano (1993) have provided several examples of using spreadsheets to enhance mathematical understanding with regards to word problems and algebra. In each of these cases the open-ended nature of spreadsheets was used to advantage so novices could effectively explore a mathematical concept. Spreadsheets have also been promoted as a valuable tool for learning statistical concepts (e.g., Arganbright, 1992; Bakeman, 1992; Piele, 1990).

On the other hand, not all researchers are enthralled with spreadsheets per se. For instance, Connell (1998) wrote:

It is not surprising that technology's role has often been to replace the labors of computation while offering little for developing student problem...spreadsheets...suffer from deficiencies when applied to classroom instruction. The problem is that they remain a black box to the elementary student with only the outcome being visible. The methods of solution which lead to this answer and rationale remain invisible to the elementary student--thus weakening their potential applicability.

Connell's critique is right on the mark. Spreadsheets offer the potential to enhance a learner's problem solving skills and conceptual understanding, but if not implemented wisely spreadsheets can easily turn into a mechanical time-saving device that does nothing to challenge and enhance student understanding. Towards this end, great care was taken to frame the instructional challenges in such a manner that the pedagogical benefits of interacting with Excel would be emphasized, while minimizing the mechanical aspects.

To the novice, spreadsheets can seem daunting. In fact spreadsheets can appear very similar to the LOGO programming language. As Excel has developed over the years it has kept all the power of its bare-bones formuladriven approach but it has also incorporated features that make the product more friendly by allowing users to easily incorporate color, buttons, sounds, and other graphical or multimedia elements into their spreadsheets. From the perspective of the doctoral students in this study, these user-friendly features were a decided bonus. However, the major pedagogical improvement to Excel over the years has been in the "guts" of the program which now allows users to name cells and arrays. The consequence of this improvement is that the resulting formulas can look and read in a more intuitive manner. Naming, along with the Excel's spreadsheet design tools, make the program much more approachable to novice users

There are five primary features that contribute to Excel's power as a pedagogical tool. Those features include: the value of spreadsheets as open-ended tools, the importance of conceptual formulas, the usefulness of intuitive naming, the ability for dual-coding, and the variety of design tools that a program like Excel provides users. Each of these five features is discussed briefly in the next section.

OPEN-ENDED TOOLS

The power of a spreadsheet program as a pedagogical tool lies in its open-endedness. The program itself gives the user nothing except a workbook filled with empty cells. This is much like the "look" of entering a programming software program or a blank piece of paper. The onus, therefore, is on the student to construct a worksheet which demonstrates their ability to develop a step-by-step way to calculate a particular measure.

Excel offers a wide variety of built-in functions that can be easily accessed either from the "insert function" command or from the "analysis toolpak" add-in. Using these two sources of support, an individual can easily have Excel calculate means, standard deviations, t-tests, and fairly complex ANOVA tests with the push of a button. However, students in the course were not allowed to use any of these functions or add-ins beyond what were defined as six core functions. The functions they were allowed to use included: Sum, Average, Count, CountA, DevSq, and Sqrt. All other calculations had to be built on the foundation of these core functions. It also should be noted that the DevSq function (which would automatically calculate the sum of squares for an array of numbers) was not introduced until about 30% of the way through the course. Thus this function was provided to students as a time-saving device for calculating complex ANOVA designs after they had already demonstrated the ability to calculate sum of squa res using the more basic commands.

While Excel had many advantages from the point-of-view of the instructor, this open-endedness needed to be sold to students as a benefit because the program appears so ugly initially. Students needed to understand that Excel doesn't "do" but it "makes." Towards this end students were provided with examples of what Excel can do through some simple instructor-made learning playgrounds, The basic learning playground structure they saw was similar in design to the ANOVA playground in Figure 1. There are a few essential elements of this sample playground that needed to be incorporated into the subsequent student-made learning playgrounds. First, the purpose of a playground is to serve a pedagogical end so that the data set entered should not be very large. A small data set allows novices to better understand the contribution of each score to the final measure or test. Second, a learning playground provides the user with both an analytic pathline and a graph. The analytic pathline is simply a way of making the step s involved in calculating a measure transparent to the user. The graph is used as a complementary way to help students visualize some aspect of the measure. Third, users were encouraged to engage in the learning playground by changing the raw scores. In the case of Figure 1 there were challenges which were included on a separate piece of paper. For example one challenge asked students if they could create a set of scores involving three groups in which there is "a significant Between groups effect with [eta.sup.2] between 0.15 and 0.25 and the homogeneity of variances assumption is met." By having a model of a basic learning playground students were better able to conceptualize how Excel might be used to create meaningful products.

Conceptual Formulas

One of the conundrums in basic statistics texts has been how to present formulas. Through algebraic transformations the same formula can have very different looks. While these algebraic transformations may rightly be seen as relatively trivial to an expert, the novice sees these different formats as being entirely different concepts. More importantly, some equation formats are more intuitive to students because they are directly connected to the concept. Many transformed formulas have become standard use because they historically represented an easier way to calculate the concept through the use of a calculator or by hand. While the days of hand calculation or basic calculators have gone, some of these raw score formulas continue to be used. Shavelson (1996) provided an excellent discussion of deviation versus raw score formulas as well as showing how the two are algebraically equivalent (p. 108).

Let's consider an example. A fundamental concept in statistics (and especially central to the course in this study) is that of sum of squares. This measure represents the sum of all the squared deviation scores in a group. The idea of sum of squares (or SS) is used extensively in ANOVA tests, which essentially take advantage of different methods for partitioning the total sums of squares into separate components (e.g., [SS.sub.within] and [SS.sub.between] a simple one-way ANOVA).

The raw score formula for SS (Shavelson, 1996) is not linked to the conceptual idea. In fact this formula does not include the mean. Yet, for beginning students, they understand that deviation scores are central to calculating SS. Thus for a novice, the representation for a deviation score should show up somewhere in the SS formula. Alternatively, the deviation score formula is much more intuitive and meaningful to students. The formula says that SS is the result of summing up all of the squared deviation scores in a group. The key is that the notion of "deviation scores" and their relationship to SS are built into the formula itself.

Admittedly if one were stranded on an island with only a seventies-style calculator and a large group of numbers then the raw score approach would be easier to calculate than the deviation score approach. However, in a spreadsheet program it is easy to incorporate the deviation score formula that reinforces the conceptual structure of SS.

Intuitive Naming

The power of spreadsheets lies in the ability to write user-created formulas that can calculate a wide variety of mathematical measures. For the novice, however, it can appear that they are learning two languages at once: statistical formulas and spreadsheet language. The newer versions of spreadsheets have helped to bridge this gap through the ability to name cells and arrays.

Only a few years ago the formula for the mean of a group of scores would have looked something like "=average(A4:A12)." With the newer versions of Excel, however, a student could have named the A4:A12 array as "girls" and so the subsequent formula for the mean of this group of scores would be "=average(girls)". This ability to name individual cells or groups of cells (arrays) results in more intuitive formula creation by students.

Dual Coding

Many novices have problems "seeing" or visualizing key statistical concepts. It's often helpful for students to have a graph that allows them to get a better sense of what's going on with the data. A typical example would be the use of a scatter plot to get a visual snapshot of whether there is a relationship (linear or otherwise) between two variables. In this study the most useful graph for students was a simple pie graph which could dynamically display the partitioned sum of squares in a specific ANOVA design. While graphs themselves have been used productively for a long time, spreadsheets offer the possibility of dynamic graphs which automatically change as the raw scores are altered. This provides the user with quick visual information about how the change in a score contributes to changes in the bottom-line measures of interest such as the partitioned sum of squares.

There is increasing research support for the idea that learners better understand material when there is an effective combination of text (or numbers), sound, and images (e.g., Mayer, 2001 for an extensive discussion). In a modest way the students in this course sought to take advantage of the link between analytic numbers and a visual representation to create better learning.

Design Tools

Finally, the fifth area that contributes to the power of Excel lies in its design tools. At its simplest level a spreadsheet program can consist only of cells and the ability to write formulas. One of the advantages of Excel is the great variety of educationally-relevant design tools that it offers the user. Those design tools include pop-up notes, buttons, URL links, inclusion of movies and sound, accounting arrows, and other tools. Each of these tools provides flexibility and power to a student's ability to create an educational product using Excel.

CONTEXT AND PARTICIPANTS

All of the participants in this study were students in an intermediate-level doctoral education course in statistics. The students had already taken general courses in research methods and introductory statistics. None of the students had any significant previous experience using the Excel program. Of the eight students, three were K-6 grade level teachers, one a high school teacher, and four were higher education instructors (in management, nursing, and the social sciences). Seven of the students were female and one male.

The use of a spreadsheet program for creating learning playgrounds was new to all of the participants. Two of the participants had seen instructor-made learning playgrounds in a previous introductory statistics course. However, in the first class all participants were familiarized with the instructor's set of prebuilt playgrounds and the ways in which the expectations of the course were different from simply emulating the instructor's past work.

This study describes how students' understanding, as evidenced by their final learning playgrounds, demonstrated conceptual growth. All students completed four basic learning playgrounds and two special learning playgrounds over the period of one semester. The challenge of creating a learning playground was straightforward: "imagine you are creating a learning product for first-year doctoral students in a beginning level statistics course." The context for this challenge was given in their syllabus as:

Statistics, like great philosophical storytelling, is essentially about high-powered exciting ideas. In a beginning statistics course it is primarily the instructor that does the storytelling. In this course you will be equally responsible for the storytelling thorough the creation and sharing of learning playgrounds. The major course outcome is the development of a spreadsheet portfolio demonstrating through story, graphs, and spreadsheet savvy your understanding of the core ANOVA techniques. You will choose two of the four ANOVA spreadsheets to re-package as an effective electronic learning tool for future students. For this outcome you can stretch your own creativity, educational expertise, and attention to detail to create two compelling learning experiences for future students. In the final class session you will have the opportunity to present and explain one of your e-learning products to your colleagues.

While students in the course had many questions about the final products, they had a basic idea of what could be accomplished through the instructor-developed model playgrounds. It was equally clear to them that the expectation was that their products would be more fully-rounded than the "bare bones" playgrounds developed by the instructor.

The biggest problem for the instructor was developing an instructional format that would allow students to learn key skills in Excel without taking up an excessive amount of in-class time. Towards that end the following components of the course were structured to support student learning of Excel: classroom support, student guide and CD, and e-mail. Each of those components is briefly described below.

Classroom. General motivation was not a problem with this small group of students. However, most of them were not computer savvy, and none were Excel savvy. This provided a challenge in terms of structuring the course: after all the course was about ANOVA designs, not about computers. The first class meeting was used as a three-hour workshop on how to use Excel. In addition, approximately 20-30 minutes was spent during each of the subsequent 4.5 hour class meetings to address Excel questions and provide instruction on more refined Excel skills. In this way the instructor hoped to balance the needs of students for in-class Excel support while maintaining a primary focus on learning new statistical techniques.

Guide and CD. While many Excellent books exist about how to learn Excel, none of them addressed the key features of Excel that students needed to learn aligned to the specific challenges students faced in the course. To help remedy that problem the instructor created a 49-page guide to Excel specifically designed to address the core spreadsheet skills they would need. In addition, students were presented with a CD at the beginning of the course that contained QuickTime movies showing students how to do specific procedures in Excel. QuickTime movies allowed the instructor to show a procedure rather than write about it. The CD also contained additional support materials.

48 hours and e-mail. Students were told that if they e-mailed the instructor with a question they would receive an answer within 48 hours, but that usually they would receive a response within 24 hours. While some Excel questions could be adequately addressed by way of text, others necessitated creating additional QuickTime screen movies which could be sent as an e-mail attachments to students. Through these support mechanisms it was hoped that complete computer novices in the class would be provided with adequate support in learning Excel.

LEARNING PLAYGROUNDS

Students made four learning playgrounds and then selected two of these to "make special." The idea behind the "special" playgrounds was that it gave students a culminating opportunity to bring to bear the full scope of their statistical knowledge as well as their full abilities in working with Excel to create an effective learning playground product. For instance, their original one-way ANOVA learning playground was made early in the semester when their understanding of key ANOVA concepts and their skills with Excel were at the lowest. By being able to "make special" their one-way ANOVA playground later in the course, students were given the opportunity to demonstrate a refined understanding of both the statistical technique and a more skilled use of employing Excel to create a learning environment.

The four learning playgrounds they needed to create were a one-way ANOVA, factorial ANOVA, repeated measures ANOVA, and split-plot (one within and one between factor) ANOVA. Student choices for the special playgrounds were evenly distributed amongst the four techniques. Each student presented one of their special learning playgrounds at the last class meeting. The presentations were audio recorded. About three months later the instructor interviewed three of the students to get a better idea of student perceptions of the course.

A content analysis of materials was conducted by reviewing students' earlier work in the course, then looking at each students' two special learning playgrounds, relistening to their verbal presentations, and reviewing the three recorded interviews. Three key factors emerged as benefits of using an active-learning approach in the statistics classroom. First, students' working knowledge of statistical techniques was deeper than in previous classes. A second benefit was that students became active reflective thinkers about how one can best learn statistical concepts. The third benefit was that students incorporated multiple modes of learning into their final products. The last two benefits (reflective thinking and multiple modes) are not directly beneficial to statistical understanding per se but were highly motivating to the group of students in this study since they all shared an interest in education and the educational process. Each of these three benefits of constructing learning playgrounds is discussed i n the following section.

Deeper Understanding

One of the key benefits of developing learning playground products was that student understanding of key concepts appears to have been much more thorough. This was reflected in five specific ways: (a) they showed a greater connection to research design, (b) their technical and conceptual understanding was integrated, (c) their interpretive abilities increased, (d) they benefitted from dual coding of sum of squares partitioning, and (e) they asked better questions in class.

Connection to research design. This was probably the outstanding difference between previous classes and the active-learning one. Students were constantly trying to link new statistical techniques with potential research designs. An unintended benefit of the learning playground challenge was that students initially chose to create one context which they could use across the various statistical techniques. Below are two sample contexts. Each student would then alter the basic context, as appropriate, so that it fit the demands of the specific statistical technique under study.

Example Context 1: Twelve students (six females & six males) were in a control group (received lecture and reading material on Arterial Blood Gas (ABG) analysis) and twelve students (six females & six males) were in an experimental (treatment) group (received lecture, reading material, and were assigned to participate in an electronic ABG learning activity). These undergraduate nursing students were enrolled in an Introduction to Pathophysiology course at a private University in Northern California.

Example Context 2: You decide to investigate the effects of cooperative learning on your fifth grade students' achievement in social studies. Three table groups worked cooperatively on reading the textbook and answering questions. Three other table groups worked individually on the same materials. At the end of the two-week unit you administered a test on the social studies unit to determine which method had been most effective.

While adapting one "master context" to the various ANOVA techniques makes sense in terms of saving time, it had the important pedagogical benefit of prompting students to think more deeply about the relationship between context and statistical technique. The end result was that students were more savvy in terms of thinking about design first, then statistical technique to match that design. In Figure 2 part of a student's work is displayed that is teaching a future user about how to think through creating an ANOVA analysis. From the start, this student begins with the theme of thinking about research design.

Technical and conceptual merged. Among novices there tends to be a wide gulf between technical and conceptual understanding of a statistical technique. Typically some students will be better at retaining a technical understanding as reflected in an ability to remember and use appropriate formulas. On the other hand, other students will be better at retaining knowledge of the conceptual basis and implications of a statistical technique. For example, a student may be fairly good at remembering what a basic regression analysis does and what it may tell us but not know how to interpret the constant and slope numbers. In general there is a trend amongst faculty to emphasize conceptual understanding over technical competence in many statistics courses because of the pervasive influence of personal computers and statistical analysis packages. The intended benefit of using the learning playground challenge in this course was that students' conceptual understanding of ANOVA designs would be increased. The unintended r esult of the challenge, however, was that students had a much better understanding of both the conceptual and technical underpinnings of ANOVA. More importantly, they generally saw how one "type" of understanding increased and supported the other "type." The end-of-semester student presentations emphasized this link, and how for them being able to blend the technical with the conceptual made the ANOVA concepts both stronger and easier to understand for the students.

Figure 3 shows one example of how a student took the time to provide meticulous conceptual and technical support in a learning playground. This figure captures the portion of the learning playground which contains the ANOVA table for a repeated-measures design. Through text boxes, and hidden pop-up notes, the student was able to guide a user though how to think about both the conceptual and technical information contained just in this one small portion of the product.

In Figure 4 another student approached this issue m a different way. Here they've chosen to make explicit the technical underpinnings of the calculations in the ANOVA table. Although Figure 4 only shows a portion of one sheet, you can see there is conceptual support (in terms of the "statistical significance" and the "effect size guidelines" balloons) while if you scroll down the sheet you'll also find more detailed support regarding specific calculations.

Interpretive abilities increased. Interpretation implies the "What does it mean?" question in statistics. This was easier for students to address because students created their own context for a learning playground. Interpretation was an area in which many students previously felt "shaky." Consequently it was an issue everyone addressed in their products even though it was not an explicit standard for a learning playground. Many students also chose to include questions in their final products, which would prod the user to think more deeply about statistical analysis. In Figure 5 you see part of a learning playground where the student used a steady stream of questions (with an accompanying "answer" sheet) to keep users thinking more deeply about what was going on in terms of design and analysis.

Dual coding of partitioning. Students were required to include a graph in their learning playground. However, many students went beyond this simple requirement as they were acutely aware of the benefits of dual coding. Specifically they spoke about the benefits of being able to enter data, or change data, and immediately see the impact those changes had on a pie or interaction graph. A number of students placed their graph right next to the data entry area. As a user enters, or changes, data they get immediate feedback on how that new number impacts the resulting partitioning of the sum of squares. Since all of the learning playgrounds used small sample sizes (typically 5-10 subjects per cell of a design) it was also much easier for a user to see how changing one score might impact the subsequent partitioning.

Better questions in class. Increased student attention to detail and "making connections" in the learning playground products also resulted in better classroom questions. This was true even for topics which were not the focus of any student product. Not only did students ask more questions than in previous versions of the course, they also tended to ask better questions. Many of those questions focused on the linkage between statistical analysis and research design or on the linkage between conceptual importance and the technical structure of a particular ANOVA design.

Reflective Thinking

During their inclass presentations most students. said, "When I thought about what it was like for me when I was taking my first course in statistics...." These students took advantage of their educator's expertise, and interest, to reflect on their own strengths and limitations as novice learners of statistics to try to develop products that would explicitly meet the gaps they felt existed in their own knowledge base. Perhaps just as important they also paid attention to motivational factors. They realized that for them, and others, that first course in statistics was often a struggle motivationally and cognitively. As a result of their reflections on how we learn there were three general themes that emerged in their products: (a) valuing the importance of context, (b) valuing the importance of ongoing questions, and (c) linking the technical with the conceptual.

The importance of context. In reflecting on how they and others learn, students found it quite important that their learning playground products include problems in contexts that their audience would readily understand and relate to. They also realized the value of having a consistent context and general problem, so that only specific questions changed with different statistical techniques but an overarching general context was maintained.

The importance of questions. Students reflected on the relatively passive nature of most statistical learning. Specifically, they could imagine their own learning playground products easily becoming very passive unless they built in some form of interactivity. Students did this in two ways. First, most built in a series of questions which the user was supposed to answer as they progressed through a learning playground. A second general "questioning" device was the use of a challenge to get the user to interact with the basic data entry area. All learning playground products had a way in which they encouraged the user to change the raw data and make sense of the resulting changes in statistical analysis. Some challenges were more specific than others because they asked the user to try to create a raw data set that would result in a prespecified statistical result (e.g., resulting in a nonsignificant difference for both independent variables but a significant finding for the interaction effect in a 2-way ANOVA) .

Linking technical with conceptual. As mentioned earlier, students reflected on the intimate link between conceptual idea and technical procedure. Most learning playground products tried to explicitly combine both factors. In Figure 6 part of a worksheet is displayed for a one-way ANOVA learning playground. On this sheet the student is trying to communicate both the "big idea" of sum of squares, and help the user link this idea to the practical ability to calculate each SS. Not seen in Figure 6 is how the sheet scrolls down and provides the user with a practical example of calculating each SS, then it moves back towards the conceptual by linking the calculations with substantive meaning.

Multiple Modes

Perhaps more than anything else, students thought about the nature of the learning process itself. They were conscious that most people are not motivated to take statistics courses (especially doctoral education students who may not have taken a mathematics course for over 20 years). While paying attention to statistical learning by itself, students were also savvy about including elements of instructional design that would be generally helpful regardless of the content being learned.

Design sensitivity and color coding. All students were aware of the importance of creating a design that was attractive for users. For some students this meant trying to simplify their learning playground products as much as possible. Often simplification was achieved by "hiding" information through devices such as pop-up notes so the user could get information and support on a need-to-know basis but was not overwhelmed with information when first navigating a learning playground. Others focused less on simplification by itself, but tried to make the layout of their product appealing and organized so that a user would feel motivated to progress through the learning playground. Figure 7 shows the beginning sheet for one such learning playground. While the design of this sheet is relatively complex, it makes great use of organizational features that would help the user progress through the sheet in a helpful manner: notice the step-boxes on the left side with arrows pointing to relevant features. Even at the ve ry beginning there is the use of a question card (far right). On the far upper left there is a toolbox icon: if the user clicks on this icon they are hyper-linked back to the "main page" of this student's learning playground product.

A number of students also made conscious use of color coding. While all students used colors to make their products look better, some went further and used color coding to make their learning playground visually intuitive to a user. For instance, one student always colored the cells for the first independent variable with one color, for a second independent variable with another. A simple device, but one that allowed the user to easily make sense of the contributions of each variable to the statistical analysis.

Schema theory. Students also tried to take advantage of how people tend to organize categories of objects in their minds. Specifically, several students took advantage of schema theory to help learners better understand the relationships between concepts. Figure 8 shows the beginning of one such learning playground. This particular student used colored figures to represent the different "players" in their product. [SS.sub.within] was represented by a red figure in Figure 8, his "brother" [SS.sub.between] was represented by a blue figure. Later the user meets their cousins, d(Cohen's d) and [eta.sup.2]. Then there's the cranky old uncle, F-test, who "just sits around and judges things." By presenting these various statistical entities as part of a connected "family" of concepts, and by using an appropriate level of humor, the student created a memorable learning playground that succeeded in emphasizing both the conceptual, and the technical.

Interactivity. Another way to engage a user was by incorporating elements that would make the learning playground products more interactive. As mentioned earlier, one such device was the use of questions within the products. Another way to add interactivity was through the use of buttons. One student became particularly engaged with the potential benefits of buttons. A problem with Excel as a educational tool for end-users is that most people are unfamiliar with using spreadsheet software and it's various capabilities. The value of buttons is they provide a way of incorporating some of the more advanced features of spreadsheets without requiring the user to know how to access those features: press a button and the event happens! For instance, in Figure 9 buttons are used to help a user access Excel's ability to show "trace precedents" and "trace dependents" arrows. A user can select any cell in the sheet and then press the "Show Before" (or trace precedents) button to see which cells (or numbers) contributed to the calculation of the number in the cell selected. In Figure 9 the arrows are coming from four arrays of numbers (representing the four cells in a 2x2 factorial design). Press the "Show After" (or trace dependents) button and the user sees what subsequent calculations the selected cell contributes to. Again, in Figure 9 the [SS-.sub.within] cell is shown to contribute to [MS.sub.within] and the three observed F-test numbers.

Visual/analytic dual coding. This feature was discussed earlier, but students saw it as crucial that a user see how key statistical measures change as the raw data is altered. The natural interactive ability of Excel was used in that this spreadsheet program automatically and immediately updates a graph (and other subsequent calculations) when a data point is changed.

Storyline as building block. The idea of storytelling was central to most learning playground products. There were two basic techniques students used for storytelling, and most students used both techniques simultaneously. The first technique was to create a context that the user could understand. The second technique was to name worksheets within a workbook to create a systematic way for the user to sensibly progress through a learning playground product. A few students also used hyperlinks so the user could click on an icon and be taken to a relevant web site or other worksheet. The organization used by students was typical of a story structure: introduction, playground (or body), more details, and a conclusion.

Building their own products. A number of students went one step further by challenging the user to create their own spreadsheet calculations. They predicted that a very active, hands-on mode of learning was going to most effective. Figure 10 shows the beginning instructions that one student provided their user for how to name cells and arrays.

LIMITATIONS

Overall the learning activities worked very well, resulting in a higher level of student learning relative to previous versions of the course given by the same instructor. Nonetheless, there were limitations to the learning playground challenge that could be improved in the future. These limitations were understood from informal student comments and from three student interviews conducted a few months after the course had ended. The key areas for improvement included: a critical time zone for learning Excel, using more of a "studio" approach, and making stronger connections between Excel and statistical analysis software.

Critical Zone

The course was about ANOVA techniques, not about computers by themselves. Excel served a powerful role as a pedagogical tool for helping students enhance their understanding of key concepts. However for Excel to function in a supportive role, the basics of Excel need to be learned by students quickly and well. Student comments indicated two key issues regarding learning Excel: (a) there is a 2-3 week window in which they are willing to live with the ambiguity of not knowing how to use Excel competently and (b) early on they need to have tangible evidence that they are making good progress as novice Excel users. Put differently, learning "refined" Excel skills as the course progresses is fine, but the core Excel skills need to be mastered by students in a relatively short amount of time (three weeks or less). Most important to note is that it is students who feel this need for relatively quick mastery of basics. Related to this critical learning zone of three weeks are the connected issues of a hands-on worksh op, debugging skills, and the student guide.

Hands-on workshop. Students were provided with an initial three hour workshop in a computer lab about learning Excel. While the workshop was useful, student comments indicated that the workshop was not as hands-on as needed. In short, there was too much looking at an overhead screen about how to do it rather than the students themselves trying out the software. In the future the initial three hour workshop will be revised so that it is more hands-on.

Debugging skills. For many students Excel is like learning a programming language. One implication is that to learn Excel partly involves having good "debugging" skills. However, especially at the beginning, students felt ill equipped to deal with situations in which Excel did not work as advertised. Put differently, they learned from trial-and-error and from e-mailing the instructor, but they thought an up-front explicit tutorial on debugging Excel would be very useful for future novice learners.

Student guide. Students were given an instructor-written 49-page guide called Excel to the Rescue! The guide provided information on all the key spreadsheet skills they would need, in the order in which they would use them, accompanied by lots of screen-shot pictures showing exactly what the procedure would look like on the computer screen. The guide was supplemented by some instructor-made QuickTime movies showing what to do in Excel. Despite the benefits of this guide and accompanying movies, the guide did have some typos regarding specific statistical procedures. These were easy to correct through e-mail and in class, but for novice learners (again in the first three weeks) it was sometimes hard for them to discern whether a problem they were experiencing was due to a typo or their perceived low level of competence in Excel.

Studio Approach

As the course progressed some student products were shared with the class to highlight various ideas or approaches to designing a learning playground. However the only thorough opportunity students had to see and hear about the thinking behind each other's playgrounds was at the last class session. Many students commented that they would have loved to have seen other students' work "in progress" as the course went along. They noted that other people had very good ideas and by sharing ideas and experiments it would have spurred them to make even richer and better learning playground products. In other words, students commented that something like a "studio session" as a small part of each class where two students share their work-in-progress would be a great addition to future courses. Such a studio session could be done in 20 minutes (out of a 4.5 hour class meeting) and seems to be a reasonable change to make.

Making Connections

While students made many connections between ANOVA concepts, statistical formulae, and research design, some mentioned that they lacked the mechanical connection between the results presented in an Excel learning playground and the output from SPSS. Since SPSS is prevalent at many institutions, it was a reasonable request to integrate the Excel-SPSS connection into a future version of the student guide.

CONCLUSIONS

The overall result of using Excel as a pedagogical tool for students to create learning playground products appeared to be a rich and powerful way for students to learn statistics at the intermediate level. One student described the general experience succinctly:

Most meaningful was that I was able to understand the statistical concept behind the actual assignment when I used the computer. It was probably the first time in statistics I was able to go, "Aha, I understand." I used everything: the text, the class notes, the presentations in class. But then I went to the computer and did the hands-on part. It was then that I was able to put it all together in a way that I could understand.

In past versions of the course there were always clearly designed presentations, a useful text, class notes, and the such. The unique added feature to this particular course was the inclusion of Excel as a pedagogical tool. Another student aptly described the value of going back and forth between conceptual understanding and mechanical implementation:

I think the best thing (about the course) was that in order to make the playgrounds I really had to understand what ANOVA was in general, as well as the particular ANOVA we were working on. Plugging in equations was busy work, but it was crucial that I had a solid conceptual understanding before I started creating the playground. Then when I was working through it I understood the concept more and more. So at the end, when we had to come up with a scenario that fit with the design, that was Excellent because I really had to think about it. Then when we had to address what it all really meant in practical terms--that really solidified everything!

This quote speaks not just to the general value of Excel as a learning tool, but specifically to the importance of engaging in a process where the learner is going back and forth between the conceptual and the technical since each of these different types of understanding reinforce, and deepen, the overall learning experience of the student. One student highlighted another key feature of the learning playground products:

I was able to integrate creativity and a great deal of fun into something I originally perceived as dry and uninteresting. It made it a real exercise in design and how I present the material. Also the way you framed it--that it would be an instructional product--that made me think carefully about novices and wanting to help them.

So, in addition to the pure statistical learning features, the "fun" features of the learning playground products was crucial in the long run. The course did challenge learners to use their educational expertise and to use a modicum of design sensitivity to create learning playground products that were appealing and effective. Perhaps the most important potential aspect of the course was revealed in an e-mail message one student sent to the instructor after the end of the course:

I think the most important thing that I learned from the class was that I could be good at things I had more or less written off. I considered myself pretty poor at statistics and a lot worse at Excel (ignorant, to say the least)! At some point during the class I realized that I was quite good at designing ANOVA playgrounds. How amazing!!! Learning the discrete skills was, of course, important. But the most valuable thing was learning that I could be good at just about anything given excellent instruction and the opportunity to approach the subject from a position of strength (my creative pedagogic skills). The wonderful thing about teaching teachers is that there are also second order benefits--I'll find ways to incorporate what I've learned into teaching others. So you've taught a lot more people through me.

While it is unrealistic to hope that a course would have a similar impact on all students, it is encouraging that some students see the long-term benefits of creating and making worthy products as part of the learning process.

FUTURE DIRECTIONS

This investigation yielded positive results supporting the notion that active-learning environments can be effective, but there is still much to learn. This study was different from the Mitchell (1997) study where the course focused on learning computer technology. Instead this study more closely resembled the needs of a typical instructor: using technology as a pedagogical tool which supports learning but is not the focus of the learning process.

There are some areas of practical concern for instructors that this study does not address. All the students in this intermediate level statistics course already had taken courses in introductory statistics and introductory research design. How would such a course work with true novices in a beginning level introductory course? In addition, the course had the luxury of containing only eight students. What problems would be incurred in adapting this course to the more realistic situation of an introductory doctoral-level statistics course where the enrollment is typically 25-30 students? In the near future this approach will be tried out in an introductory statistics class. Hopefully the lessons learned from this initial experiment into a technology-supportive role for active-learning can be modified to meet the even greater needs of novices grappling with introductory level statistical concepts.

IMPLICATIONS FOR OTHER FIELDS

This study looked at only one field of study: statistics. What implications are there for other fields--either in mathematics or the sciences in general? One of the general problems in mathematics and science education has been that so few students are attracted to them. In the sciences especially there has been some attention paid to the developmenet of Problem-Based Learning (PBL) curricula that challenge students to be more active learners. With the advent of more intuitive software such as Excel, some multimedia programs, and specialty mathematics/science software (such as the Geometric Sketchpad) there are even greater oppotunities now for students to use such computer based tools to actively construct meaning from their learning experiences. The essence of the challenge used in this study ("Can you teach others about the concept you just learned?") is the kind of challenge that could be used in many other fields of study. The specific advantages to instructors in mathematics and science is that such an approach seems to help students learn the material better while also allowing them to develop a deeper, more positive relationship with the content. These positive cognitive and motivational benefits seem to indicate that pursuing this general "teach-to-learn" approach may have a fairly wide applicability.

APPENDIX

STATISTICAL TERMS

Sum of Squares: This is a measure of all the squared deviation scores in a sample of numbers. The statistical measures of variance and standard deviation are derived from this basic measure. The general notion of sum of squares gained even greater importance in the early 20th Century with Ronald Fischer's insight that the total sum of squares for a sample could be partitioned into independent pieces.

ANOVA: A basic one-way ANOVA is one way of partitioning the total sum of squares into two independent pieces: sum of squares within (variation we can't explain) versus sum of squares between (variation which can be explained due to systematic differences between the levels of the independent variable).

F-Test: The F-test (named after Ronald Fisher) is used to assess if there is a statistically significant difference between the levels of the independent variable. The test essentially takes advantage of the partitioned sum of squares to create a ratio that is then used to assess statistical significance.

Eta-squared: A measure of effect size. In a one-way ANOVA it is calculated by dividing the sum of squares between by the sum of squares total. The resulting number gives us the percentage of the total sum of squares that can be explained by between group differences. In more complicated ANOVA designs, eta-squared can be calculated by using the sum of squares for a specific effect diviided by the sum of squares total.

References

Abramovich, S. (1995). Technology-motivated teaching of advanced topics in discrete mathematics. Journal of Computers in Mathematics and Science Teaching, 14(3), 391-418.

Abramovich, S., & Nabors, W. (1998). Enactive approach to word problems in a computer environment enhances mathematical learning for teachers. Journal of Computers in Mathematics and Science Teaching, 17(2/3), 161-180

Arganbright, D.E. (1992). Using spreadsheets in introductory statistics. Statistics for the twenty-first century (pp. 226-242). Washington, DC: Mathematical Association of America.

Bakeman, R. (1992). Understanding social science statistics: A spreadsheet approach. Hillsdale, NJ: Lawrence Erlbaum.

Bargh, J.A., & Schul, Y. (1980). On the cognitive benefits of teaching. Journal of Educational Psychology, 72, 593-604.

Benware, C.A., & Deci, E.L. (1984). Quality of learning with an active versus passive motivational set. American Educational Research Journal, 21, 755-765.

Brophy, J., & Alleman, J. (1991). Activities as instructional tools: A framework for analysis and evaluation. Educational Researcher, 20(4), 9-23.

Bruner, J. (1966). Toward a theory of instruction. Cambridge, MA: Harvard University Press.

Connell, M. (1998). Technology in constructivist mathematics classrooms. Journal of Computers in Mathematics and Science Teaching, 17(4), 311-338.

Dewey, J. (1963). Experience and education. New York: Collier Books.

Dugdale, S. (1998). A spreadsheet investigation of sequences and series for middle grades through precalculus. Journal of Computers in Mathematics and Science Teaching, 17(2/3), 203-222.

Kafai, Y. (1995). Minds in play: Computer game design as a context for children's learning. Hillsdale, NJ: Lawrence Erlbaum.

Mayer, R. (2001). Multimedia learning. Cambridge, England: Cambridge University Press.

Mitchell, M. (1993). Situational interest: It's multifaceted structure in the secondary school mathematics classroom. Journal of Educational Psychology, 85, 427-439.

Mitchell, M. (1997). The use of spreadsheets for constructing statistical understanding. Journal of Computers in Mathematics and Science Teaching, 16(2/3), 201-222.

Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York: Basic Books.

Piele, D. (1990). Introductory statistics with spreadsheets. New York: Addison-Wesley.

Rogers, C. (1969). Freedom to learn. Columbus, OH: Merrill.

Shavelson, R. (1996). Statistical reasoning for the behavioral sciences. Boston: Allyn and Bacon.

Sutherland, R., & Rojano, T. (1993). A spreadsheet approach to solving algebra problems. Journal of Mathematical Behavior, 12, 353-383.

Whitehead, A.N. (1929). The aims of education and other essays. New York: Macmillan.

Zajonc, R.B. (1960). The process of cognitive tuning in communication. Journal of Abnormal and Social Psychology, 61, 159-167.

RELATED ARTICLE: ADDITIONAL RESOURCES

An example of an instructor-made learning playground and a variety of student-made playgrounds can be downloaded at the following web link: mitchellprion.com/constructanova