An analysis of student learning at a testing web site emphasizing descriptive chemistry.

Since 1997, the World Wide Web (WWW or Web) has been used to enhance the performance of students on the Advanced Placement (AP) exam by providing a web site dedicated to the descriptive portion of the exam. This web site provides repeatable testing with feedback in the form of quizzes produced and graded in less than one second. This study evaluated the effectiveness of such an online testing and feedback system on outcome performance as measured by user records and validated by user surveys.

The results indicate the potential for enhancing learning from instructional web sites focused on teaching with repetitive testing and feedback. Successful web teaching with repetitive testing must emphasize random, repeatable tests with a large item database and extensive feedback given with each item returned for grading. Future web teaching/testing sites should focus enhancements on three areas: (a) the user interface, (b) the tutorial and feedback components, and (c) the nature of the quizzes.

**********

For over 40 years, The College Board has offered Advanced Placement (AP) Chemistry Examinations for high school students (College, 2000). This service allows academically well-qualified high school students to remain in high school while still engaging in learning activities that are appropriately challenging and rigorous. In 1999, the College Board reported the grading of over 48,000 chemistry exams (Sheridan, 1999).

The AP chemistry exam is broken down into five content areas: (a) structure of matter, (b) states of matter, (c) reactions, (d) descriptive chemistry, and (e) laboratory. Anecdotal evidence suggests that the descriptive chemistry section of the AP chemistry exam is traditionally difficult for high school students. The exam's difficulty can be attributed to the nature of the material and the current structure of the AP curriculum. Descriptive chemistry is difficult to teach because it requires either a large amount of memorization or experience; it tends to be disjointed within the traditional curriculum.

Since 1997, the World Wide Web (WWW or Web) has been used to enhance the performance of students on the AP exam by providing a web site dedicated to the descriptive portion of the exam (Crippen, Brooks, & Abuloum, 2000). This web site provided repeatable testing with feedback in the form of quizzes produced and graded in less than one second. The questions were stored in a database, and served to the user in a format similar to that of the pencil and paper exam. The database was designed in such as way that all of the user's interactions were recorded and logged.

The database contained over 200 descriptive questions and represented all of the descriptive chemistry questions given on AP exams since 1968. The quiz items were not multiple choice. Students were expected to write a net ionic chemical equation to represent a written reaction statement. The student's answer then resembled an essay response. In practice, student responses are hand graded with the essay part of the AP exam.

Testing and the Web

The study described here is grounded in literature concerning the use of the Web for teaching/learning, and the use of repetitive testing measures. Although use of the Web for teaching/learning is becoming very common, little is known about its effectiveness. Much is known about the effectiveness of repetitive testing with feedback, however.

Using the Web to deliver instruction is becoming common practice. Colleges and universities are looking seriously at their roles in providing instruction over the Web (Jones, 1995). Entire degree programs are now available over the Internet.

Using the Web as a teaching tool appears to be as effective as is the traditional classroom. The use of web instruction has shown no significant difference in test scores when compared to traditional test scores (Wegner, Holloway, & Garton, 1999). Web instruction does seem to impact student impressions of the course in positive ways, however (Brown, 1995; Wegner et al., 1999).

In the simplest form, computers are able to process repetitive operations continuously. This makes them ideal for providing students with practice scenarios and problems. Computers can provide problems, evaluate responses to those problems, and provide immediate feedback. Providing appropriate feedback is known to increase the rate of learning (Lhyle & Kulhavy, 1987). As students practice with concepts, whether in the form of homework problems, in-class assignments, or with quizzes and tests, they learn concepts. Practice helps make the use of concepts automatic (Posner & Snyder, 1975; Shiffrin & Schneider, 1977).

Feedback from practice need not be confined to the correctness of an answer. Kulhavy and Stock (1989) described effective feedback as potentially providing the learner with two types of information: verification and elaboration. Verification distinguishes correct answers; elaboration is information providing the learner with clues to distinguish the correct answer. Since feedback points out missing or misunderstood knowledge, it provides an ideal teaching moment. Thus, any feedback that further explains what students do not know adds to the value of the feedback.

Effective practice must occur with feedback. Feedback that explains how answers are determined and provides further instruction is most effective (Pressley & McCormick, 1995; Renkl, 1998). Many experts believe that technology's role in learning is providing feedback while allowing time for reflection and revision (Bransford, Brown, & Cocking, 1999)

Practice with feedback promotes self-regulated learning. Pintrich described self-regulated learning as "involving the active, goal-directed, self-control of behavior, motivation, and cognition for academic tasks by individual students" (Pintrich, 1995). Brooks, Nolan, & Gallagher advocated the promotion of self-regulation through the inclusion of specific strategies in web-based instructional design (2001).

While practice typically implies homework or some sort of classroom work, testing also can serve as practice. Practice tests and quizzes not only prepare students for the type and variety of questions on a final graded exam, they also can serve as a teaching/learning tool for course content (Foos & Fisher, 1988).

Studies show that the Web facilitates the delivery of tests and supports distance learning (Shiffler, 1996). Integral to the delivery of online courses is a testing component. Many support the notion that the role of schools in the future is linked not to the information that they provide, but the role they take in assessment (Jones, 1995).

While not a panacea, web testing shows great promise in many areas. The success of web testing may hinge on providing effective feedback. The nature of the feedback ensures that the student not only identifies what they do not know, but has the opportunity to reflect and acquire missing knowledge.

Full scale web teaching requires that all materials and testing be delivered over the Web. Current practice reveals that web testing is being implemented on a more narrow scale and in selective situations.

Using the Web to grade homework is a very practical application of the technology. Online homework systems have shown to have a positive effect on student test scores (Penn, Nedeff, & Gozdzik, 2000). This is especially so when the exam questions require the use of memorized concepts or the use of simple problem solving algorithms (Paull, Jacob, & Herrick, 1999).

Using a web-based homework system has shown to increase exam scores by 5-10% and increase student self-confidence (Penn et al., 2000) in organic chemistry. Penn et al. showed a logistic increase in practice exam score as the number of attempts increased.

Parallel to web-based homework systems are web-based practice systems where students can take practice quizzes of their own volition. While these practice exams do not affect a student's grade, early research suggested that students who use such practice score higher on actual exams than do students who use only traditional study methods (Hall, Pilant, & Strader, 1999). Charlesworth (2001) reports a direct correlation between online practice quiz scores and traditional exam scores for general chemistry students.

The purpose of this study was to evaluate the effectiveness of such an online testing and feedback system on outcome performance as measured by user records and validated by user surveys. This study was undertaken to elucidate the manner in which effective teaching and learning can be conducted using the Web. The following three research questions guided the investigation:

1. Do users learn descriptive chemistry from using the AP descriptive chemistry web site?

2. Do self-reported AP students perceive the site to be effective in learning descriptive chemistry?

3. Are the students' perceptions of effectiveness in learning descriptive chemistry and their use patterns at the descriptive chemistry web site consistent with the literature on the effectiveness of repetitive testing?

METHOD

The design of the AP descriptive chemistry web site emphasized tracking of an individual's use and surveying user perceptions. This design yielded information about the effectiveness of the site as a learning tool. The site's design was consistent with the suggestions and considerations of many experts in providing online instruction for high school science students (Hoffman & Others, 1997; Lyons & Others, 1997; Ritchie & Hoffman, 1997). In addition, this site was designed with the understanding that incentives positively affect student learning and performance (Gettinger, 1989).

Participants

The research sample represents individuals interested in descriptive chemistry, especially as it applies to the AP chemistry examination. High school students were intended subjects. The sample was recruited by word of mouth, conference presentations, listserve postings (CHEM-ED-L, AP-CHEM), and newsgroup postings (misc.education.science, k12.education.science). The final sample included 404 self-reported AP students.

Database Structure

Study participants were identified by the e-mail address they provided. Once informed consent was given, each new user had the option of completing a preliminary survey. By completing this survey, each user established a login. Each time a user logged in, all of his/her transactions were recorded together with the e-mail login. The access time, access address (computer IP number), specific identify of the items sent, responses made, and tutoring requested were all recorded.

Procedure

Each user was asked to complete a brief demographic survey the first time they used the site. By completing this survey, each user established a logon. Each subsequent visit to the site required the user to login and bypassed the survey.

The database used the logon to create a unique record for each user. This record was accessible to the user; it documented their use of the site. From the record, each user could review previous quizzes as well as their use of tutoring.

Once logged in, students requested a quiz. This quiz was a randomly generated set of eight items. The items were drawn from 14 categories defined arbitrarily by one of the authors.

The content of most quizzes was similar to that used by The College Board. This was accomplished by assigning a chemistry topic category to each quiz item. When a quiz was constructed, each of the eight items came from a different category. Weight was given to the categories based upon the total number of items available from that category. The item database grows each year that an AP examination is given; so, the quiz evolves with the AP test-questioning scheme. While probability of a duplicate quiz was extremely low, the likelihood of seeing items repeated, though small at first, grew with each test.

The user had the option of having tutoring sent with the items. This tutoring was in the form of web pages written for each category of items. Having tutoring with the items allowed the user to learn more about the category of reaction before submitting an item response for grading. The database tracked which items and tutoring were sent to each user, and this information was included in their record. The database also tracked which tutoring was used with each item.

The user also had the option of choosing to have an entire quiz constructed from one of nine predefined item categories. Students could request a content topic, such as electrolysis, and then have their entire eight-item quiz chosen from that topic area. By choosing typical AP Quiz, however, users received a randomized quiz similar to what they would receive at an AP examination. This feature allowed users to hone their skills on specifics with their choice of similar environment, or prepare for the randomness of the actual AP examination.

Following the 2000 AP Chemistry Examination, the raw data was removed from the database. The e-mail addresses were inspected, and obvious false addresses were removed.

A survey designed specifically for the AP students was sent on Tuesday, May 16, 2000, the day of the AP chemistry exam. The survey instrument was an e-mail response survey composed primarily of Likert scale responses. This method was chosen over other options because of its potential range and ease of use for the respondents (Bowers, 1998). The surveys were strictly text in nature and consistent with design standards currently reported in the literature (Bowers, 1998; Gaddis, 1998). Respondents simply selected reply in their e-mail program, added their responses to the original message, and then selected send to return the survey to the researchers.

Data Analysis

Statistics used in demographic descriptions are as they appear; date, time, gender, and so forth. A tutoring request implied that a user requested some form of extra information not supplied in the quiz itself and was counted by subcategory. The nature and use of quizzes however, require the use and explanation of some special statistics. The use of such statistics allows for a thorough documentation of learning at the descriptive chemistry web site. A quiz sent by the database was not necessarily completed or returned by the user. Thus, the emphasis of the learning analysis focuses upon those quizzes returned for grading. Quiz numbers where students did not attempt any items were removed before analysis.

Quizzes returned for grading were given a unique number and broken down item by item. Each of eight items contained six potential data fields for answers. If any of the six data fields for a particular item contained text, the item was counted as an attempt at that item. An individual item score was computed for each item of every quiz. Scores for completed items were combined to compute the Average Score per Completed Item statistic. This statistic is a numerical average including only those items attempted by the student.

Quiz items were further classified as new or as repeat. A new item is one from the database that the user had never seen before. A repeat item is one that appeared on a previous quiz the user has submitted for grading. Thus, the average score per item completed statistic can be further sub-classified by item status. In addition, the value and significance of repeat items were quantified.

RESULTS

The AP student population was split 50% male, 45% female, and 5% not reported. The largest percentage of AP students reported their age as less than 18 years (84%). The students were located throughout the United States with the largest concentration in the northeastern US (Figure 1).

More than one third (41%) of the AP students reported using the descriptive chemistry web site during chemistry class. Ninety-seven percent of the self-reported AP students who responded to a post AP chemistry examination survey reported having taken the AP chemistry examination on May 16, 2000.

AP Student Site Use Pattern

The self-reported AP students averaged 21.2 logons to the web site and averaged 7.7 tutoring requests per individual over the 1999-2000 academic term. Acid/base chemistry was the most requested tutoring component by a sizable margin.

Over the 1999-2000 academic term, the AP student group requested an average of 4.7 quizzes. The quizzes were mostly of the typical category, indicating the quizzes were similar in item composition to the actual ALP chemistry examination.

AP Student Learning Analysis

Learning at the site is documented using the following statistics: average raw quiz score, average score per completed item, average score per new item, average score per repeat item, date, day number in the study, and quiz number. The average raw quiz score per quiz number and the average score per completed item (sub-classified as overall, new, and repeat) all increase m a linear fashion with high correlation coefficients. The average raw quiz score increased as the study progressed on a day-by-day basis. Significance testing was not applied to the general AP student population as the numbers of students attempting any quiz ranges from 300 students attempting one quiz to a single student making an attempt after 56 quizzes.

AP students did not complete all of the items on every quiz. Many quizzes were returned partially completed. However, no pattern emerged for the number of items completed. Raw quiz score increases linearly as the quiz number increases, r = +0.873.

This pattern suggests that AP students earned progressively higher quiz scores as they took more quizzes. However, the inconsistent nature of the number of quiz items completed suggests that the raw quiz score is not an appropriate statistic for documenting learning.

In addition, due to the size and random nature of the database, it was possible for students to have attempted and completed repeat quiz items only. This would cause the raw score to rise, but not necessarily imply the acquisition of new knowledge. Instead, it might imply an acquired ability to answer specific test items.

Following the first quiz, the number of repeat items attempted on each successive quiz grows linearly as the number of new items decreases linearly. Beyond quiz number 28, all of the quiz items were essentially repeat questions. Completing only repeat items could account for the raw score increase. Interestingly, the AP students did not attempt all of the repeat items for every quiz. The percentage of repeat items attempted per quiz hovers around 80% until quiz 35 where it climbs to essentially 100% as the number of quiz attempts falls.

A plot of the average score per completed item for all 56 quizzes attempted by members of the AP student group reveals a linear relationship, r = +0.852. This indicates a steady improvement over time. The average score per repeat item is also linear with quiz number, r = +0.775.

Yet, the change in average score per completed item is lower for the repeat items. While a portion of the linear increase in the overall average score per completed item can be attributed to the repeat items, some additional factor also was improving the scores.

As further evidence of improved performance and learning through using the site, the average raw score was plotted as a function of the day in the study. The average raw score increases in a positive fashion as the study progressed through the 1999-2000 academic term.

The data show an increasing trend. This trend coupled with a linear increase in the average score per completed item gives strong evidence for learning by the AP students as they request, complete, and are provided with feedback on graded quizzes.

The AP Special Group

The AP Special Group represents a 55-member subset of the AP student population that took a minimum of seven unique quizzes and for which significance testing was applied. This subset represents a group of students that made extensive use of the site for taking quizzes. This subset was chosen to further elucidate and identify characteristics of a population intent on using the web site as a means for achieving success on the AP chemistry examination.

The demographics and descriptive use statistics for the AP Special Group were consistent with those of the general AP student population. However, exceptions included:

* One hundred percent of the AP special group who responded to a post AP chemistry examination survey reported having taken the AP chemistry examination on May 16, 2000.

* The AP special group averaged 62.5 logons to the web site, more than four times that of the remaining AP student population.

* The AP special group averaged 20.1 tutoring requests, three and one-half times that of the remaining AP student population.

AP Special Learning Analysis

For the AP special students, raw quiz score increases linearly with quiz number at a rate close to that of the entire AP student group. This suggests that the AP special group accounts for a large portion of the gains documented by the entire self-reported AP student population.

The average score per completed item as a function of quiz number for the AP special student group reveals linear relationships for repeat items and overall scores consistent with those reported for the general AP student population. An analysis of the first seven quizzes for the AP special students was intended to quantify the effect of new items on the linear increase of average score per completed item in the small quiz number range where the student score is most influenced by new items.

Within the first seven quizzes, students attempted a proportion of new and repeat items, allowing for separation and quantification of the impact of these variables on the overall average score per item completed. Because the AP special group represented a sizable, stable sample, significance tests were applied to all correlations for the first seven quizzes.

The correlation between the average score per completed item overall and quiz number is significant, r = +0.871, n = 55, p<0.005, one tail. The correlation between the average score per new item completed and quiz number is significant, r = +0.631, n = 55, p<0.005, one tail. The correlation between the average score per repeat item completed and quiz number is not statistically significant, r = +0.218. The rates of change in average score per new item is close to 1.5 times that of the average score per repeat item, 0.0422 and 0.0282 points per item respectively. However, the average rate of change overall is greater, 0.0594 points per item. Table 1 summarizes the correlation analysis for the first quizzes.

An item analysis of the first seven quizzes indicates that the percentage of new items attempted starts at 100% on the first quiz and decreases approximately 5% with each successive quiz. The repeat items attempted start at 0% on the first quiz and increase proportionally. The large percentage of new items attempted in the first seven quizzes means that the overall average score per completed item is largely dependent upon the average score per new item completed. With increasing quiz number, the dependence of the overall average score per completed item shifts progressively from the new items attempted to the repeat items attempted.

The combined linear nature of both the average score per new item completed and the average score per repeat item completed increases the linear nature of the overall average score per item completed. The overall rate of change for the average score per item is a weighted total of the percentage of item types attempted and their average score per completed item value.

The dependence of the overall average on the new item average suggests that the linear increase of overall average score per item completed in the first quizzes can be attributed to a continually improving performance on new items, those never seen before. This increasing positive change is most likely attributable to an increase in knowledge.

Student Perceptions of Site Effectiveness

Thirty percent of the AP student population with valid e-mail addresses responded to the post exam survey (105 students). When asked "How helpful was the descriptive AP chemistry quiz site in preparing you for the AP chemistry examination?" the students that responded generally felt that the site was helpful. However, certain circumstances such as computer failure, access issues, and reported learning style conflicts caused some students to feel the site was less than helpful. AP special students felt that the site was very helpful. Their comments are universally positive and affirm the design intentions of the site.

Sample student free responses:

* "The site gave me a better idea of what would be expected of me and how the test would look."

* "Though the format for the reaction section changed, the site was still invaluable for practice. Some of the same problems appeared on the exam."

* "Since I wasn't able to cover some areas in my class, I was learning about stuff from the web site."

* "It was helpful with the reactions part of the AP if you actually sat and did some of the questions. It taught you skills."

When asked, "Did using this site improve your performance on the AP chemistry examination?" the students that responded affirmed this site as helping to improve their performance on the AP chemistry examination. Students cited problems entering equations with a computer keyboard, time requirements, and the lack of knowledge about their actual exam scores as reasons for the site not improving their performance on the AP chemistry examination. One hundred percent of the AP special population that responded to the survey felt using the site improved their performance on the AP chemistry examination.

Sample student free responses:

* "This site improved my score I think....Because it had so many questions to practice and helped you learn what kind of reaction you were doing. It was very useful."

* "I learned what I needed to study."

* "It definitely improved my ability to perform the reactions. Without it, I probably wouldn't have known what to look for."

DISCUSSION

Self-reported AP students used the AP descriptive web site with the explicit purpose of improving either their performance on quizzes or their knowledge of the concepts needed to answer the descriptive questions on the AP chemistry examination. This is supported both by their use of the tutoring components and the quizzing/feedback components at the site. The AP students were not casual observers of the site; their use implies intent to learn descriptive chemistry.

The fact that the AP students used the site during the school day and reported use during chemistry class provides further evidence to support their learning intent. Yet, the majority of students recorded their use in just a short time period before the AP chemistry examination was given. While the students have used the site at effective times in the day and under supportive circumstances, the fact that they access the site just before the AP chemistry examination limits their potential achievement.

Learning by the AP students at the descriptive chemistry web site is documented using a number of statistics. These statistics include an increase in the average raw quiz score per day during the study. While the AP students do show a linear improvement in quiz score with quiz number, they never acquire proficiency. The linear increase in the average raw quiz score with increasing quiz number suggests that the greater the number of quizzes attempted by students, the better the overall score. However, the students start out earning very low quiz scores, and few students achieve a perfect score.

The AP students show a linear increase in the average score per completed item through the first nine quizzes in all item sub-classifications. However, though the students improve in all sub-categories, their total improvement is about one-half of one point per item from a baseline average of one point per item. At this rate, the students should have achieved proficiency after 29 quizzes.

Some students selected only certain items to attempt: particular chemistry content, new items, or repeat items. Their rationale for this is unclear. No clear pattern exists for any group's completion rates for types of items. Whether or not an item was completed appears to be a random occurrence.

The total size of the AP student group permitted selection of subsets. The criterion for inclusion in one subset was extensive use of the feedback component at the site. This group, named the AP special group, consisted of 55 self-reported AP students who returned the largest number of graded quizzes.

Self-reported AP students who responded to the post AP examination survey generally felt that the site was helpful. However, circumstances such as computer failure, access difficulties, and reported learning style conflicts caused some students to feel the site was less than helpful. This survey feedback supports and is consistent with the learning analysis based on the group's use of the site.

The majority of the AP students affirmed the AP descriptive chemistry web site as helpful in improving their performance on the AP chemistry examination. Students cited problems entering equations with a computer keyboard, time requirements, and the lack of knowledge about their actual score on the May 16, 2000, AP chemistry examination as reasons for the site not improving their performance on the AP chemistry examination. This feedback supports and is consistent with the learning analysis based on the group's use of the site.

If the statements made by the AP students responding to the post AP examination survey concerning both entry errors and computer failures are valid for the entire user population, then such errors and failure could affect the quiz scores. These errors could account for variability in compiling certain statistics, for example.

The NP special group is a subset of the self-reported AP student group who made extensive use of the quizzing/feedback components of the AP descriptive chemistry web site. The AP special group had the highest use of both the tutoring components and the largest number of quizzes attempted. The AP special group was dedicated to using the NP descriptive web site to learn descriptive chemistry and to improve their score on the AP chemistry examination.

A date and time analysis for the NP special group suggests that this group had similar use patterns to those of the general NP student group. These included logon times, use during the week, and dates during the academic term. The principal distinctions in use for the NP special group were the total number of logons and the consistency of returning graded quizzes.

The AP special group also shows a linear increase in the raw quiz score with quiz number. The minimum, maximum, and average change in score are consistent with those for the general NP population. Again, these scores start low and never reach the maximum score. Although the AP special group makes the most extensive use of all the resources at the AP descriptive web site, few, like the general AP population, ever achieve a perfect score.

Analysis of average score per item with quiz number again is very similar to the general AP population. Statistical significance is shown for the increase in average score per new item completed and in average score per item completed overall for the first seven quizzes. The NP special students start with a higher average score-per-item on the first seven quizzes, but improve an average of 0.0594 points per item. The NP special group scores higher on early quizzes, improves their subsequent scores incrementally, but the rate of increase is very low. At their initial rate, the AP special group should have achieved proficiency after 36 quizzes.

These results suggest that AP students will benefit from taking as many quizzes as possible. The fact that the AP students in this study increase their scores at such a slow rate suggests that some other mitigating factor be required to enhance the effectiveness of the site. However, analysis of a select group of three AP special students reveals an increase in quiz score over several weeks or months. This implies that an early use of the site may influence both the maximum score earned and the rate of score improvement.

Self-reported AP students selected as the AP special group who responded to the post AP examination survey felt that the site was helpful. Their comments are universally positive, and affirm the design intentions of the site. All of the survey respondents (100%) felt using the site improved their performance on the AP chemistry examination. This supports and is consistent with the learning analysis based on the group's use of the site.

Recommendations

The results of this study indicate the potential for enhancing learning from instructional web sites focused on teaching with repetitive testing and feedback. Successful web teaching with repetitive testing must emphasize random, repeatable tests with a large item database and extensive feedback given with each item returned for grading. In this way, the web teaching site focuses users on the content to be learned and supports user self-regulation.

This study merits several recommendations for further study. Future web teaching/testing sites should focus enhancements on three areas: (a) the user interface, (b) the tutorial and feedback components, and (c) the nature of the quizzes.

The user interface should give users many options in how their quizzes are constructed. In addition to choosing the types of items they receive, users should have the option of using pull down menus to aid in formula writing, or to have simple hints like the type of question included with the question. Instead of creating a new quiz when the New Quiz button is selected, the user could be presented with a series of switches that allow them greater flexibility in quiz design.

Early research indicates that computer practice and handwritten testing are not congruent (Russell, 1999; Russell & Haney, 1997). Many of the survey respondents in the current study indicated problems with formula entry at the web site. For practice sites where the ultimate measure remains a handwritten exam, site users should have the option to vary the amount of formula typing they are forced to perform in their quizzes by selecting pull down menus as an option. Flexibility in the user interface that allows users to tailor their quizzes eliminates bias in the scoring due to entry problems and better identifies techniques of successful users.

Enhancing tutorial components presents the best opportunity for continued research while increasing the rate of learning for the users. Depth and breadth should be added to the content tutoring associated with each question. In addition, the feedback provided with each graded response should be item specific. Item-specific feedback includes remarks consistent with each unique item in the database. In this way, users have the option to take tutoring about reaction types; yet, each graded response returns remarks consistent with the individual items on their quiz.

With the greater flexibility in quiz design provided to the user, they not only have the option to take content tutoring, but they can turn the item-specific tutoring off and on. Item-specific tutoring addresses the low rate of learning in this study and allows the researcher to identify successful learning strategies.

Recommendations from the results of this study have the potential to assist web teachers, researchers, and students. Web teachers should provide repetitive testing components with feedback at their sites to maximize student learning. Further research at the descriptive chemistry web site should include enhancements to the user interface, the tutoring components, and the nature of the quizzes. Including these enhancements could provide valuable data concerning successful use patterns and further assist users in learning the content emphasized at the site.

[FIGURE 2 OMITTED]

Table 1

Correlation Analysis Summary for Average Score Per Item Completed by
Quiz Number for the First Quizzes

Group                   Overall                        New Items
              Rate       BL        Corr     Rate       BL       Corr

AP Student
(9 quizzes)  0.0756    0.8095    +0.935    0.0551    0.8491    +0.889
AP Special
(7 quizzes)  0.0594    0.8928    +0.871 *  0.0422    0.9243    +0.631 *

Group                 Repeat Items
               Rate      BL       Corr

AP Student
(9 quizzes)   0.041    1.3339    +0.551
AP Special
(7 quizzes)   0.0282   1.3818    +0.218

Rate = points per item per quiz

BL = Baseline (points)

Corr = Pearson correlation

* p<0.005, one tail

References

Bowers, D.K. (1998). FAQ's on online research. Marketing Research, 10(4), 45-49.

Bransford, J.D., Brown, A.L., & Cocking, R.R. (Eds.), How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.

Brooks, D.W., Nolan, D., & Gallagher, S. (2001). Web teaching (2nd ed.). Norwell, MA: Plenum.

Brown, A. (1995). Evaluation of teaching and learning processes in a computer-supported mechanical engineering course. Computers & Education, 25(1), 59-65.

Charlesworth, P. (2000). WebCT@MTU: Rejuvenating general chemistry. Paper presented at the Biennial Conference on Chemical Education, Ann Arbor, MI.

College Board (1999). AP Chemistry: Distribution of grades for all 1999 APChemistry candidates. Retrieved from the World Wide Web May 23, 2002 from: http://www.collegeboard.com/ap/students/chemistry/ grade99.html

College Board (2000). AP technical corner: From humble beginnings. Retrieved from the World Wide Web May 21, 2002 from: http://www.collegeboard.org/ap/techman/chap1/

Crippen, K.J., Brooks, D.W., & Abuloum, A. (2000). A web site supporting the AP descriptive chemistry question. Journal of Chemical Education, 77(8), 1087-1088.

Foos, P.W., & Fisher, R.P. (1988). Using tests as learning opportunities. Journal of Educational Psychology, 80, 179-183.

Gaddis, S.E. (1998). How to design online surveys. Training and Development, 52(6), 67-72.

Gettinger, M. (1989). Effects of maximizing time spent and minimizing time needed for learning on pupil achievement. Educational Research Journal, 26, 73-91.

Hall, R.J., Pilant, M.S., & Strader, R.A. (1999). The impact of web-based instruction on performance in an applied statistics course. Paper presented at the International Conference on Mathematics/Science Education and Technology, San Antonio, TX.

Hoffman, J.L., Kupperman, J. & Wallace, R. (1997). Online learning materials for the science classroom: Design methodology and implementation. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago.

Jones, D.P. (1995). Higher education and high technology: A case for joint action. Denver, CO: Western Governor's Association.

Kulhavy, R.W., & Stock, W.A. (1989). Feedback in written instruction: The place of response certitude. Educational Psychology Review, 1(4), 279-308.

Lhyle, K.C., & Kulhavy, R.W. (1987). Feedback processing and error correction. Journal of Educational Psychology, 79, 320-322.

Lyons, D.J., Hoffman, J.L. Krajcik, J.S., & Soloway, E. (1997, March). An investigation of the use of the World Wide Web for online inquiry in a science classroom. Paper presented at the Annual Meeting of the National Association for Research in Science Teaching, Oak Brook, IL.

Paull, T.A., Jacob, M.J., & Herrick, R.J. (1999). Automated homework in electrical engineering technology. Paper presented at the American Society for Engineering Education symposium, Charlotte, NC. Retrieved from the World Wide Web May 23, 2002 from: http://www.asee.org/conferences/search/99conf34.PDF 

Penn, J.H., Nedeff, V.M., & Gozdzik, G. (2000). Organic chemistry and the Internet: A web-based approach to homework and testing using the WE_LEARN system. Journal of Chemical Education, 77(2), 227-231.

Pintrich, P.R. (1995). Understanding self-regulated learning. In P.R. Pintrich (Ed.), Understanding Self-Regulated Learning (pp. 3-12). San Francisco: Jossey-Bass.

Posner, M., & Snyder, C. (1975). Attention and cognitive control. In R. Solso (Ed.), Information processing and cognition. Hillsdale, NJ: Erlbaum.

Pressley, M., & McCormick, C.B. (1995). Advanced educational psychology for educators, researchers, and policymakers. New York: Harper Collins.

Renkl, A. (1998). Cognitive science: Learning from worked-out examples: A study on individual differences. Retrieved from the World Wide Web May 21, 2002 from: http://www.umich.edu/%7Ecogsci/abstract/598renkl.html

Ritchie, D.C., & Hoffman, B. (1997). Using instructional design principles to amplify learning on the World Wide Web. (ERIC Document Reproduction Service No. ED415835)

Russell, M. (1999). Testing writing on computers: A follow-up study comparing performance on computer and on paper. Educational Policy Analysis Archives, 7(20).

Russell, M., & Haney, W. (1997). Testing writing on computers: An experiment comparing student performance on tests conducted via computer and via paper-and-pencil. Educational Policy Analysis Archives, 5(3). Retrieved from the World Wide Web May 21, 2002 from: http://olam.ed.asu.edu/epaa/v5n3.html

Shiffler, N. (1996). The state of the Web in the web of the states. Paper presented at the Annual meeting of the National Council on Measurement in Education, New York.

Shiffrin, R.M., & Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. Psychological Review, 84, 127-190.

Wegner, S.B., Holloway, K.C., & Garton, E.M. (1999). The effects of Internet-based Instruction on student learning. Journal of Asynchronous Learning Networks, 3(2). Retrieved from the World Wide Web May 21, 2002 from: http://www.aln.org/alnweb/journal/Vol3_issue2/Wegner.htm