Considering the efficacy of web-based worked examples in introductory chemistry.

Theory suggests that studying worked examples and engaging in self-explanation will improve learning and problem solving. A growing body of evidence supports the use of web-based assessments for improving undergraduate performance in traditional large enrollment courses. This article describes a study designed to investigate these techniques in a practical web-based setting. This study tracked introductory college chemistry students' use of a course quizzing system that provided both worked examples and strategy suggestions embedded within the quiz items. Student perceptions of the effectiveness of the enhancements were measured and correlated to use and performance. The findings indicate students made use of both the worked examples and self-explanation prompts, felt they were helpful, and the interventions improved performance.

**********

This study examined the potential for web-based worked examples and self-explanation strategies as part of a formative assessment system in introductory chemistry. The study is grounded in the literature on the use of worked examples and self-explanation strategies applied to a web-based repeatable testing environment. Theoretically, each strategy, considered separately, is well supported. This effort aimed to combine the three strategies in the chemistry domain in an effort to improve performance and problem-solving ability for undergraduate students in an introductory course.

The intent of this article is to: (a) provide a theoretical rationale, (b) describe our assessment system, (c) describe student use of the system, (d) report student response to the system, and (e) correlate student use with performance indicators. The reported data support the use of similar techniques for engaging students in using web-based worked examples and self-explanation prompts. Additional information regarding this study, including access to the assessment system and copies of our survey instruments, are available through the Web (http://crippen.nevada.edu/chemistry/WE_Study/).

THEORETICAL FRAMEWORK

Problem solving is an integral component of mathematics and science instruction (National Science Standards, 2003; Principles and Standards, 2003). Developing problem-solving ability is complex, however. Traditional approaches focus on teaching students algorithmic procedures and heuristic techniques. Algorithms are different from heuristics. Algorithms are a specific set of procedures that lead to a predefined outcome. In problem solving, algorithms always generate correct answers. Heuristics are collections of techniques or "rules of thumb" for developing a solution. Heuristics may contain algorithms.

The following is typical of traditional instruction in both science and math courses. The teacher presents a problem or concept by way of a problem. These problems are usually well-structured, multi-step in nature, require conceptual and procedural knowledge, and are presented with a model answer. The teacher articulates and models a solution algorithm or heuristic to aid students in solving similar problems. The teacher's presentation is followed shortly with an opportunity for students to practice the problem solving demonstrated in the form of an assignment. The assignment likely includes multiple versions of similar problems. The students are left to their own devices to complete the assignment, often times with some of the problem solutions provided by the back of their textbook.

While these strategies may be successful on some level, they certainly are not efficient. Learning and reiterating an algorithm or heuristic should not be confused with understanding a concept or possessing problem-solving ability. Traditional approaches that focus on skill instruction followed by voluminous practice tend to be lacking in skill transfer, conceptual understanding, and improvement in problem-solving ability (Pressley & McCormick, 1995). Additionally, research suggests that worked examples are more appropriate for inexperienced learners while problem-solving practice is more appropriate for experienced students (Kalyuga, Chandler, Tuovinen, & Sweller, 2001).

We conjecture that providing worked examples and self-explanation prompts within test items in a web-based repeatable testing environment has the potential to improve problem-solving ability and conceptual understanding. This is accomplished by giving students the opportunity to develop their own problem-solving strategies as a result of focusing their attention on problem states and problem-solving operators.

Worked Examples

Worked examples are detailed problem solutions that contain identifiable qualities and characteristics (Ward & Sweller, 1990). These representations are constructed in such a way as to provide the learner with some structure for understanding how the solution was established without providing a script or algorithm (Atkinson, Derry, Renkl, & Wortham, 2000). Depending upon the topic area, these solutions can take many forms. Snippets of computer program code can be effective worked examples for computer programming while structural diagrams work well for architecture and engineering. Worked examples have been shown successful in computer programming, algebra, and geometry instruction (Carroll, 1994; Paas & van Merrienboer, 1994).

From a cognitive perspective, studying worked examples lessens the load on working memory while focusing student attention on problem states and problem-solving operators (moves) (Sweller, 1988, 1994). Students who have been engaged in studying worked examples as an instructional strategy adopt problem-solving techniques quicker and have improved problem-solving performance (Chandler & Sweller, 1991; Cooper & Sweller, 1987; Sweller & Cooper, 1985; Tarmizi & Sweller, 1988). This typically manifests itself as instruction that is more efficient and yet improves student performance on problems requiring transferred knowledge and skills.

Using worked examples to improve problem solving in science instruction is promising (Taconis, Ferguson-Hessler, & Broekkamp, 2001). The Schaum's Outline Series has remained popular over decades (Schaum's Outlines, 2003). Engaging students is the key, however. In a traditional classroom, this can be accomplished with a teacher, student groups, or tutoring, with each functioning as a direct point of contact. These techniques are not easily replicated with the Web and simply providing web-based worked examples is not likely to be successful (Smith & Jacobs, 2002).

Self Explanation

Self-explanation is a form of self-talk where a learner engages in an iterative personal dialog while engaged in problem solving. This dialog aids the learner in identifying problem states and potential solution moves. Good problem solvers exhibit a larger volume of focused self-explanations than do novices and poor problem solvers (Chi, Bassok, Lewis, Reimann, & Glaser, 1989). This form of self-regulation is intricately tied to successful problem solving (Bielaczyc, Pirolli, & Brown, 1995).

Engaging in self-explanation complements studying worked examples. Together, both strategies are useful for improving student problem solving. In fact, students who engage in self explanation of worked examples have access to those examples for creative problem-solving strategies (Didierjean, Cauzinille, & Marmeche, 1997).

Web-Based Repeatable Testing

Web-based repeatable testing is an instructional strategy that improves learning (Dufresne, Mestre, Hart, & Rath, 2002; Penn & Nedeff, 2000). While use of this strategy produces performance gains, learning rates may not be ideal (Crippen & Brooks, 2002). In fact, design issues may play a larger role than previously anticipated (Brooks & Crippen, 2001).

The web-based repeatable testing systems in use today employ a simple feedback model where a learner's response generates performance related feedback. This feedback is typically an acknowledgement of the correctness of the response and suggestions for improvement. This is an electronic application of the classic model where instruction is followed by large amounts of individualized practice. The basic model operates on the assumption that the learner is engaged with the system and is modifying their understanding accordingly. Experience suggests that this method, while successful on some level, is likely inefficient and may not improve problem-solving ability. However, it does teach students to solve problems.

Our interest is in developing students who are better problem solvers. Our approach has been to investigate improvements to our current assessment system by including worked examples and opportunities for self-explanation. This research emphasizes formative assessment where students are provided with research supported techniques embedded in traditional assessment items. The goal, consistent with national efforts, is to improve instructional efficiency, student performance, and problem-solving ability using web-based efforts.

THE STUDY

The aim of this study was to determine the potential for worked examples and self-explanation prompts within a web-based testing environment. Our primary interest was in determining whether students would use these modifications and subsequently find them helpful.

The study was designed to answer the following research questions:

* Given the option, would students use worked examples and solution strategies delivered with weekly quiz items?

* Does the use of web-based worked examples differ from use of solution strategies?

* If students used worked examples and solution strategies delivered with weekly quiz items, would they find them effective for improving their performance and knowledge of chemistry?

* Does the use of web-based worked examples and solution strategies improve student quiz scores?

An assessment system was created to include hypertext links for worked examples and solution suggestions within the assessment items. All of the assessment pieces, including worked examples and solution suggestions, were delivered from a web-accessible database. Individualized student tracking was possible by way of each student's unique login. Student use of the system, including the worked examples and solution suggestions, was monitored. Further, these students were surveyed regarding their perceptions of the effectiveness of the system and its components. Finally, use of the worked examples and solution suggestions was correlated to the students' cumulative quiz score.

The course for this study was Chemistry 115--General Chemistry 1 for science majors. It is usually offered in two sections each semester. It is a large introductory lecture course with a laboratory component. A different instructor typically teaches each section. This approach was used by one of the instructors. Enrollments typically total about 120 students per section and are limited by the size of the auditorium used for lecture. The course met twice weekly for lecture. The quizzing system described accounted for 5% of the student's overall grade. This study and the data described were collected over two consecutive semesters (Spring/Fall 2002).

Ours is a sample of convenience and our research method was quasi-experimental. The students in our experimental section were assigned by the campus course information system. Participants were given the option of having their data included in the study. Regardless of their inclusion in our study, all students in the experimental sections had access to our assessment system and learning aids we provided.

METHODS

The Assessment System

In our system, students had access to a content quiz for one week (only web-accessible). During this period, students could modify their responses at any time as their skills and understanding of the material changed. In fact, we encouraged them to change their answers. The quizzes were graded, with performance feedback given, at the end of the week. Principally, these assessments were designed as a learning opportunity. Students were evaluated, but the instructional objective was to use the assessments as a mechanism for engaging students in strategies for improving their problem-solving ability and conceptual understanding.

Our system provided worked examples and self-explanation prompts ("solution suggestions") as components embedded in weekly quiz items (Figure 1). The impetus for this is based upon the notion that combining problems with examples is better than examples alone (Stark, Gruber, Renkl, & Mandl, 2000). In fact, finding a mechanism to engage students in the examples was as much of an instructional challenge as was creating the examples themselves.

Worked examples were only provided for multi-step, well-structured problems (Figure 2). Students could study the examples at any time once the quiz became available. This included both in preparing their weekly quiz, following lecture, and after the quiz had been graded.

The solution strategies provided were designed to engage students in self-explanation (Bielaczyc et al., 1995). Generally, these suggestions provided a framework and encouragement for students to create their own solution algorithms (Figure 3). Once completed and graded, the quiz results, worked examples, and solution strategies remained available through the end of the semester.

RESULTS

Use of the Assessment System

Student use of the quizzing system, including worked examples and solution strategies, is defined based upon each interaction or "hit" on a web-accessible database. These interactions imply intent on the students' part, but do not necessarily indicate any sort of engagement. A hit indicates that a student requested a specific piece of information and that information was sent to their web browser. The database was scripted in such a way as to code a wealth of information regarding each hit. What the student did with the information is entirely unknown. The only evidence regarding student use of the information is self-reported.

Student use of the system was very consistent in both semesters (Table 1). The average hits per student was greater in the Spring semester (189) than the Fall (162), which can be accounted for primarily in the use of viewing worked examples.

The heaviest use of the system was on Monday, the day before each quiz was due for grading (Figure 4). The data indicates that students used Mondays to view the weekly quiz, view the worked examples, and to complete their quiz for grading. A pattern of viewing quizzes and examples steadily increased throughout the week. On Tuesdays, student activity focused upon viewing quizzes and to some extent viewing worked examples.

[FIGURE 4 OMITTED]

A pattern of student use was evident around the quiz due dates (Figure 5). A slight pattern associated with unit exam dates may be inferred, but is not strongly indicated.

[FIGURE 5 OMITTED]

Student Performance Implications

The use of worked examples and solutions strategies was intended to directly improve student quiz scores. It was assumed that these quiz scores would have a positive impact on the student's overall course grade. Our methodology did not allow us to directly measure the impact of using the worked examples and solution strategies on the student's final course grade. Our analysis does indicate that these variables are all related, however.

Weekly student quiz scores (5 pts.) were aggregated into a cumulative score (60 possible points). This cumulative quiz score was found to have a significant correlation with the use of worked examples and solution strategies (r=0.327, n=187, p<0.01, 2-tail). Though the cumulative quiz score was only designed to account for 5% of each student's overall course grade, the cumulative quiz score had a significant correlation with the raw final course score (r=0.708, n=187, p<0.01, 2-tail).

Follow-Up Survey

At the end of the term, before accessing the final quiz, students were asked to complete a short web-based survey. The survey contained a small number of Likert-type items. An opened response section followed each rating scale item. The intent of this survey was to gauge student perceptions of the effectiveness of the assessment system.

Reactions/Suggestions for the Quizzing System

Student survey responses regarding the quizzing system were universally positive. Three themes emerged in reading the open-ended student responses: (a) the system was perceived as fair, (b) the system was understood and embraced as a learning device, (c) in future versions, students requested a strong integration between their textbook, lecture, and the online quizzing system.

Example open-ended student elaborations.

* "Actually, I really like the way the system works. We have a good amount of time to work the quizzes, it's easy to change answers, and the examples given really help."

* "I thought the online quizzes were helpful, but I also believe that you need to read the book and practice to really do well in CHE 115. However, the quizzes did keep me up to date on studying the material, which I think is the real benefit of the online quizzes."

* "It was good stuff, but basically just a reminder for people who don't come to class, which didn't apply to my situation. I learned better in class."

Response to the Worked Examples/Solution Strategies for Understanding Chemistry

Students generally responded that both the worked examples and solution strategies were effective in helping them understand chemistry (Figure 6).

Students often cited specific examples as especially helpful. For example: "In molecular orbitals, I was able to see how the diagram of [F.sub.2] related to finding bond order and configuration, which helped me solve for [O.sub.2.sup.-2] and [O.sub.2.sup.-]." The topics addressed by these examples tended to match the most requested topics. Further, the items most often cited by students were written in such a way as to include an external representation intended to scaffold understanding (e.g., a solution table for stoichiometry or an Aufbau diagram for electron configurations).

Some students requested more structure within the worked examples ("Maybe you should explain the worked examples a little more, like step by step, so that the person knows exactly what to do.") Others described their problem-solving strategies ("I would double-check some answers by comparing them to the worked examples, such as with the Lewis structures. I made sure that my answer followed the same pattern.") Many students provided evidence in their responses that interacting with both the worked examples and solution strategies was impacting their thinking. For example: "Even though they are not the exact same questions they get you thinking on the right lines." or "They (solution suggestions) defiantly [sic] made it quicker to go through the question because they, like the worked examples, were good suggestions as to where to go or where to start the thought process."

Response to the Worked Examples/Solution Strategies for Improving Performance

Students had a much stronger opinion regarding the effectiveness of both the worked examples and solution strategies in helping improve their performance in chemistry than reported for improved understanding of chemistry (Figure 7).

[FIGURE 7 OMITTED]

Student open-ended elaborations when asked about performance improvements tended to focus on specific strategies they had used. Such as: "I remember one solution suggestion told me to look back at a definition and that would help me solve the problem and it did. Sometimes it is impossible to find something if you do not know where to look."

CONCLUSIONS

The results of this study indicate that given the option, students will use worked examples and solution strategies delivered with traditional items in a web-based assessment system. In fact, student use of the worked examples and solution strategies was extensive. They used them whenever they were available and likely referred to them multiple times. Further, use statistics indicted that worked examples and solution strategies were used to a similar degree.

Regarding impact on both performance and understanding, student survey respondents held positive perceptions of worked examples and solution strategies. Yet, how students used the interventions is likely to have differed. In open-ended student survey responses, students indicated that worked examples were used in answering specific questions or to reinforce techniques. Consider this student's response, "When I was studying for the exams the worked examples helped. Mostly with conversions." Conversely, students reported using the solution suggestions to develop or perfect their own techniques for solving classes of problems. This is illustrated in the following student survey response: "The solution suggestions showed me how to approach a problem in a different way. Some of the methods I saw I still use to solve the same types of problems."

The most direct impact of using the worked examples and solution strategies was each weekly quiz. The results show that use of these strategies has a statistically significant correlation to the student's cumulative quiz score. Though the content of these quizzes is in essence the same as the content of the other performance measures in the course, this study was not designed to link use of worked examples and solution strategies with the other performance measures. The high correlation of the cumulative quiz score and the final course grade suggests a high potential for transferring the effects of studying worked examples and solution strategies to other performance measures, however.

DISCUSSION

Learning aids, in the form of worked examples and self-explanation prompts (solution strategies), are viable options for improving student performance within a web-based assessment system. Including these aids with assessment items may provide a just-in-time delivery method. Students in this study used these web-based learning aids to improve their performance in introductory chemistry. Further, when asked to describe how they used the aids, student responses echoed the theoretical reasons for providing the aids. Namely, the learning aids allowed students to study an expert's solution, or were guided to develop a solution of their own. Regardless of performance gain, the student response "more worked examples" when asked how we could improve our system, is a powerful statement.

Though this study has shown our design conception and student use of the learning aids to be successful, the results also highlight problems. The intent of the learning aids was to guide students toward developing an understanding. Practically, the language required to accomplish this may have been too vague for some students. The student survey responses showed some evidence of this phenomenon.

The authors share a strong belief that the more time students spend with difficult material, the better chance they have at learning. In fact, our assessment system was conceived with the idea that given the option, students would spend more time interacting with the weekly quizzes. We anticipated students using different learning aids at different times of the week and potentially changing their quiz answers as they worked in laboratory, listened in lecture, studied the worked examples, or interacted with their peers. While these things may have occurred, they were not reflected in their use of the system. Some students logged-in early in the week, but many students waited until the last day, and possibly the last minute, to interact with the quizzes. This is becoming a well known pattern to web materials designers (Crippen, Brooks, & Abuloum, 2000) and likely continues to limit the performance of certain students.

Implications for Future Research

Our current studies intend to establish a more specific relationship between use of the worked examples and solution strategies regarding conceptual understanding and problem-solving ability. We believe both of these outcomes can be determined with modifications to the current course exams coupled with some form of item analysis.

In a very general sense, studying worked examples and engaging in solution strategies are intended to improve student self-regulation. If this is the case, these strategies will likely impact cognition in the form of improved self-explanation and motivation through improved self-efficacy. Improving self-explanation and self-efficacy will improve general problem-solving ability (transferable to other domains) and retention. Both of these outcomes will be examined as students translate through our series of chemistry courses.

1. How much energy is required to raise the temperature of 100 g of
water from room temperature (25[degrees]C) to its boiling point
(100[degrees]C)?

(Solution Strategy)(Example One)(Example Two)

    * 750 J
    * 7500 J
    * 3100 J
    * 31 kJ
    * 75 kJ

Figure 1. Quiz item with included worked examples and solution strategy.

A Worked Example

Sulfur is oxidized and oxygen is reduced in the example reaction below

Balanced Chemical  Pb[S.sub.(s)]   +  4[H.sub.2][O.sub.2(aq)]  ___>
  Equation
Oxidation States   (+2)(-2)           2(+1) 2(-1)
Action             Oxidized           Reduced
Function           Reducing Agent     Oxidizing Agent

Balanced Chemical  PbS[O.sub.4(s)]  +  4[H.sub.2][O.sub.(1)]
  Equation
Oxidation States   (+2)(+6)(-2)4       2(+1)(-2)
Action
Function

Close Window

Figure 2. A worked example.

A Suggestion

Remember polarity? Molecular polarity is the key to identifying
intermolecular forces

Review your notes on determining polarity Familiarize yourself with all
of the possible types of intermolecular forces and what causes them Now,
try to create a list of steps for identifying the type of intermolecular
forces for any given molecule.

Remember to practice on a few before test time!

Close Window

Figure 3. A self-explanation prompt (solution strategy).

Table 1 Student Use by Semester

                           Spring 2002     Fall 2002

         Participants          88              99
         Hits              16,634          16,038
         Hits/Participant     189             162
Actions
         Login              3,041(18.3%)    3,187(19.9%)
         View Quiz          4,430(26.6%)    4,193(26.1%)
         Edit Quiz          1,996(12.0%)    1,991(12.4%)
         View Example       7,167(43.1%)    6,667(41.6%)

If you used the (worked examples/solution strategies), rate their
effectiveness in helping you understand chemistry.

                                   Worked Examples  Solution Strategies

They did nothing for me.               1.9%                7%
I might have learned something.       16%                 20%
I learned a few things.               36%                 36%
I learned more than a few things.     23%                 20%
Extremely helpful.                    23%                 16%

Figure 6. Student survey response regarding the effectiveness of the
worked examples and solution strategies for helping chemistry 115
students understand chemistry.

Note: Table made from bar graph.

Acknowledgement

Support for this work was provided from an internally funded program for new faculty at the University of Nevada Las Vegas.

References

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181-214.

Bielaczyc, K., Pirolli, P. L., & Brown, A. L. (1995). Training in self-explanation and self-regulation strategies: Investigating the effects of knowledge acquisition activies on problem solving. Cognition and Instruction, 13(2), 221-252.

Brooks, D. W., & Crippen, K. J. (2001). Learning difficult content using the Web: Strategies make a difference. Journal of Science Education and Technology, 10(4), 283-285.

Carroll, W. M. (1994). Using worked examples as an instructional support in the algebra classroom. Journal of Educational Psychology, 86(3), 360-367.

Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of instruction. Cognition and Instruction, 8(4), 293-332.

Chi, M. T., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.

Cooper, G., & Sweller, J. (1987). The effects of schema acquisition and rule automation on mathematical problem-solving transfer. Journal of Educational Psychology, 79, 347-362.

Crippen, K. J., & Brooks, D. W. (2002). An analysis of student learning at a testing web site emphasizing descriptive chemistry. Journal of Computers in Mathematics and Science Teaching, 21(2), 183-201.

Crippen, K. J., Brooks, D. W., & Abuloum, A. (2000). A web-site supporting the AP descriptive chemistry question. Journal of Chemical Education, 77(8), 1087-1088.

Didierjean, A., & Cauzinille-Marmeche, E. (1997). Eliciting self-explanations improves problem solving: What processes are involved? Cahiers De Psychologie Cognitive-Current Psychology of Cognition, 16(3), 325-351.

Dufresne, R., Mestre, J., Hart, D. M., & Rath, K. A. (2002). The effect of web-based homework on test performance in large enrollment introductory physics courses. Journal of Computers in Mathematics and Science Teaching, 21(3), 229-251.

Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93(3), 579-588.

National Science Standards. (2003). Retrieved December 1, 2003, from http://books.nap.edu/html/nses/html/index.html

Paas, F. G. W. C., & van Merrienboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86(1), 122-133.

Penn, J., & Nedeff, V. M. (2000). Organic chemistry and the Internet: A web-based approach to homework and testing using the WE_LEARN System. Journal of Chemical Education, 77, 227-231.

Pressley, M., & McCormick, C. B. (1995). Mathematical problem solving. In M. Pressley & C. B. McCormick (Eds.), Advanced Educational Psychology for Educators, Researchers, and Policymakers (pp. 399-443). New York: Harper Collins.

Principles and Standards. (2003). Retrieved December 1, 2003, from http://standards.nctm.org/document/index.htm

Schaum's Outlines. (2003). Retrieved February 4, 2003, from http://books.mcgraw-hill.com/cgi-bin/pbg/schaums-home.html

Smith, B. D., & Jacobs, D. C. (2002). TextRev: A window into how general and organic chemistry students use textbook resources. Journal of Chemical Education, 80(1), 99-102.

Stark, R., Gruber, H., Renkl, A., & Mandl, H. (2000). Does the combination of worked-out examples and problem-solving tasks pay off? German Journal of Educational Psychology, 14(4), 206-218.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285.

Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4, 295-312.

Sweller, J., & Cooper, G. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognitive Instruction, 2, 59-89.

Taconis, R., Ferguson-Hessler, M. G. M., & Broekkamp, H. (2001). Teaching science problem solving: An overview of experimental work. Journal of Research in Science Teaching, 38(4), 442-468.

Tarmizi, R., & Sweller, J. (1988). Guidance during mathematical problem solving. Journal of Educational Psychology, 80, 424-436.

Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and Instruction, 7(1), 1-39.

KENT J. CRIPPEN

University of Nevada Las Vegas

USA

[e-mail]

BOYD L. EARL

University of Nevada Las Vegas

USA