Evaluation of Interactive Technologies for Chemistry Websites: Educational Materials for Organic Chemistry Web Site (EMOC).

The Educational Materials for Organic Chemistry Project (EMOC) developed Web-based learning materials, which were accessible by students enrolled in a variety of undergraduate organic chemistry courses at a major midwestern university. EMOC was developed and implemented by the third author and had three distinct goals: (a) to develop interactive learning materials, (b) to develop interactive tools for this learning, and (c) to evaluate the students' use of these materials.

This article reports the evaluation of EMOC conducted by the first and second authors. All authors jointly constructed the guidelines for the evaluation in two general areas: student attitudes towards and perceptions of the Website and students' specific learning from the Website.

Initially, the Website was analyzed, and the survey and focus group instruments were developed, piloted, and revised.

Next, surveys were given to students in two courses, and onsite focus groups of EMOC users and one nonuser were conducted. Attitudes toward interactive technologies on EMOC were determined by analyzing the students' responses on the initial and final surveys and the focus group discussions. Acid-base organic chemistry was chosen as a representative topic in order to probe the students' understanding of the chemistry taught in the courses and represented in the EMOC site. Students' knowledge of this chemistry was assessed by analyzing (a) the focus group participants' explanations of a graph of an acid/conjugate-base equilibrium system and (b) the concept maps of acid-base chemistry prepared by the focus group participants.

We found that students' attitudes towards the Website were generally favorable and that students believed that using the site helped them learn organic chemistry. However, the contribution of EMOC to students' actual understanding of acid-base chemistry, as discussed in the focus groups, was less clear. This result was probably affected by the low level of participation in the focus groups by nonusers.

PURPOSE/BACKGROUND OF EMOC

EMOC Website

The EMOC Project, developed and implemented by Abby Parrell, the third author, was designed to develop learning materials for a web site which could then be accessed by students enrolled in a variety of undergraduate organic chemistry courses. EMOC provides supplemental materials for these organic chemistry students. These supplemental materials include highly visual, dynamic, interactive, and guided-inquiry based learning tools. Some of the notable items available on the website include:

* animated infrared spectroscopy (AIRS) library;

* thermodynamic versus kinetic control tutorial;

* acid-base equilibria tutorial; and

* LectureOnline

All materials except the LectureOnline system are publicly accessible through the EMOC website (http://www.cem.msu.edu/[sim]parrill).

EMOC Goals

EMOC was designed and implemented with three distinct goals as guidelines:

1. To develop interactive learning materials that promoted conceptual learning or that allowed hands-on exploration of organic topics.

2. To develop interactive tools, such as databases, so that the Web could be used for some forms of alternative assessment and/or grade tracking.

3. To evaluate student use of the materials made available on the EMOC Website.

Evaluation Goals

We regard this evaluation as important because the design can serve as a guideline for evaluations of similar Web-based projects. Educational projects on the Web are proliferating exponentially, and evaluations of how these projects actually impact learning are sorely needed. It has proven relatively easy to assess students' attitudes toward using a technological innovation; evidence on the actual impact of the technology on learning has proven to be a much more elusive goal. We hope that this paper will serve as a guideline for evaluating both attitude and actual learning in Web-based educational projects.

In this article we report the evaluation of EMOC jointly designed by all of us but conducted by the first and second authors. We constructed three specific questions which guided the evaluation design:

1. What were the advantages and disadvantages of using EMOC to learn organic chemistry as perceived by the students?

2. Which parts of the EMOC site did the students like/dislike and why?

3. How did the EMOC site affect the students' understanding of specific topics in organic chemistry?

In the initial evaluation phase, the Website was analyzed, and the survey and focus group instruments were developed, piloted, and revised. In the final evaluation phase, surveys were given to students in two courses, and onsite focus groups of EMOC users and one non-user were conducted. Attitudes toward interactive technologies on the EMOC Website were determined by analyzing the students' responses on the initial and final surveys and the focus group discussions. Students' actual learning of organic concepts was probed in the semi-structured focus group discussions. In these focus groups, acid-base organic chemistry was chosen as a representative topic to probe the students' understanding of the chemistry taught in the courses and represented in the EMOC site. Students' knowledge of this chemistry was further assessed by analyzing the focus group participants' explanations of a graph of an acid/conjugate-base equilibrium system and by analyzing the concept maps of acid-base chemistry prepared by the focus group participants.

RELEVANT LITERATURE

Hypermedia Studies

Web-based materials are a form of hypermedia, operationally defined as an application of multi-media materials in a nonlinear context, including links to related nodes at another physical location (Halasz & Conklin, 1989; Heller, 1990; Melara, 1996). We hypothesized that EMOC was a form of multimedia that could affect the learning that occurred when students interacted with the Website. Kozma (1991) reviewed the literature on media in education, and he espoused the view that learners interact with a particular medium, with particular characteristics, in their efforts to construct knowledge. He further argued that the learners' interactions with this medium implicitly affects how learners construct that knowledge.

In a hypertext environment, a student must decide which links to follow and in what order the links are to be followed. Kozma noted that this very feature may make hypertext applications difficult to use for novice learners in a domain because the learners may not have enough knowledge of the domain to decide the order in which they need to progress through the hypertext. Therefore, this nonlinearity could adversely impact students' understanding of the domain. Therefore, we probed students' (users and nonusers) knowledge of specific domain topics, which were presented on EMOC and discussed in class, in order to investigate the impact of EMOC on students' understanding.

Heller (1990), in her review of hypermedia, described other concerns, which arise when hypermedia applications are used in education. In her review, she discussed a paper by Conklin (1987), who identified disorientation as a potential problem. Conklin stated that disorientation was composed of two factors: (a) attempting to find out where you are in the hypermedia system and (b) figuring out how to get to someplace else in the system.

Heller also discussed a paper by Jones (1987), who warned of the possibility of cognitive overload, which could occur due to the complex nature of the information structure within a hypertext environment. Jones argued that the learner can become overwhelmed in contemplating the large number of possible routes through the environment. Finally, Heller discussed a paper on the issue of motivation (Raskin, 1987), which can occur when the user does not feel enough commitment to the task to fully explore the hypermedia in order to find the desired information.

Heller (1990) argued that hypermedia may possess properties which encourage discovery learning, that is, learning in which exploration results in a better understanding of previously unknown relationships (Singer, 1977), or incidental learning, that is, learning which is unplanned and may not even relate to the instructional objectives of the task (Heller, 1990). She cautioned that the nonlinear nature of hypermedia may result in unexpected learning outcomes, perhaps richer, perhaps not. Therefore, based on these studies, we decided to collect information on (a) students' perceptions of ease/difficulty in using EMOC and (b) the various parts of the site which were most frequently used.

Lawless and Kulikowich (1996) investigated students' patterns of use within a hypertext environment, and they reported that these patterns of use depended upon (a) the domain knowledge of the students or upon (b) the students' interest in the topics encountered in the hypertext. Three patterns of hypertext navigation emerged: knowledge seekers, feature explorers, and apathetic hypertext users.

The knowledge seekers had the highest scores for content knowledge and were observed to spend most of their time on the informational cards in the Hypercard program, which was being used. The largest group of students in the study was categorized as feature explorers. These feature explorers were in the middle of the group in terms of domain knowledge and spent most of their time trying to understand how the hypertext worked and what kinds of screens the program offered rather than gathering domain-relevant information. The apathetic users group spent little time in the hypertext and scored below the group's average in domain knowledge. The authors speculated that the apathetic users "may not have possessed enough knowledge to effectively employ strategies that aid the navigational process" (Lawless & Kulikowich, 1996, p. 398).

These findings were relevant to our EMOC evaluation because of the features that EMOC offers. For example, students could (a) rotate molecules by dragging the mouse on the screen, (b) observe the effect of temperature and pH changes on concentration, or (c) observe an animated display of the vibrational modes of molecules. Without the proper background in organic chemistry, it was possible that students could explore these features of the site without constructing the intended understanding of the topics.

However, it was also possible, even probable, that a certain amount of feature exploration may have been necessary before a student could truly use the site to enhance his/her understanding of organic chemistry.

Finally, Windschitl (1998) reviewed the published papers about the Web and reported that most of the current literature simply describes various technological innovations, with rather rudimentary analyses of how students and teachers actually use the Web for teaching and learning. He concluded that there was a definite need for rigorous research on how students use the Web to learn. He noted that the Web is different from other forms of media because of its ability to serve as a 'conduit' for information. Therefore, he urged that one of the more critical questions to ask in research (or evaluation) is "In what contexts is the Web useful as an inquiry tool, and how are students learning in these contexts?" (p. 29). These are difficult questions to answer and require thoughtful and creative use of qualitative research techniques.

Our evaluation was designed to begin to address the second of Windschitl's critical questions, the issue of how students learn from the Web. Most evaluations try to capture the students' perceptions of how the innovation impacted their learning; our evaluation also tried to compare the chemistry knowledge of students who used the Web vs. students who chose not to use the Web.

Concept Map Studies

Concept mapping has been advocated as a tool to display students' understanding of a topic by Novak and members of his research group at Cornell University (Novak, 1976; Stewart, Van Kirk, & Rowell, 1979). Designed to show a student's understanding of the relationships between concepts at a given time, the map links concepts, usually in a hierarchical manner, by way of directional links with connecting words or phrases.

In 1990, the Journal of Research in Science Teaching published a special issue on concept mapping. Wandersee (1990) used the metaphor of "cartography of cognition" to describe the use of concept maps. According to Wandersee, concept mapping depends on the individual making the map identifying and relating salient concepts to a general concept. "Whether a process (e.g., precipitation), a procedure (e.g., titration), or a product (e.g., carbohydrate), concepts are what we think with in science" (p. 927). Concept maps employ linking words between concepts.

Wallace and Mintzes (1990) investigated the use of concept mapping to explore conceptual change in biology. Students in an experimental group completed a pretest and then were asked to draw concept maps of a biology topic. This group was then presented with additional instruction on the topic, and asked to draw another concept map and complete a posttest. The control group performed the same tasks except that the instruction dealt with a different topic, unrelated to the topic of the concept map and tests. The experimental group showed significant improvement in complexity and structure of the knowledge base as indicated by the concept maps. No difference was seen in the control group. This result, according to the authors, suggested that concept mapping is a valid technique for exploring students' conceptual change in biology.

Further evidence for the validity of concept mapping was provided by Markham, Mintzes, and Jones (1994). The study investigated the difference between concept maps of advanced biology majors and beginning nonmajors in biology. A card sorting task was also used to determine differences in the ways that the students assign class membership. The concept maps of the advanced majors were determined to be more complex, and differences in the complexity are reflected in the assignment to groups. The authors assert that their findings suggest that concept mapping is a sound and powerful tool for assessing conceptual change (p. 91). This finding further strengthens the basis for using concept maps in this study.

EMOC's Contribution

To summarize EMOC's position in the literature, EMOC is an interesting Web site because it is a conceptual, nonlinear network of modules which can be useful to students who may have different preferred modes of receiving/interpreting information. For example, the site can be useful to students who desire to manipulate objects/data as well as to students who profit from observing and evaluating. The site also includes both written text and symbolic representations, such as graphs and equations, which are decidedly different modes of presenting information. However, there are potential problems for students inherent in using such a non-linear network due to cognitive overload, disorientation in navigating the site, degree of motivation/commitment, inability to fully use different representations, or inadequate domain knowledge. Therefore, our evaluation attempted to investigate both attitude and actual understanding, and we hope that our work will help elucidate the role of the Web in helping students develop a meaningful understanding of organic chemistry.

DESIGN OF THE EVALUATION

In our evaluation we used (a) surveys of all the students in the courses for which EMOC was available and (b) focus group discussions with student volunteers. The surveys had been developed and piloted in the spring semester of 1998, and the focus group protocols had been developed from the analysis of the pilot survey. The final data collection occurred by an onsite visit in the summer of 1998, using CEM 143, an organic chemistry course for agriculture majors. During the on-site evaluation, we conducted two focus groups (one with two participants and one with one participant) of students who had used EMOC and one individual interview of a student who had chosen not to use EMOC. We also individually interviewed the two faculty involved in teaching the organic courses during that summer session.

Survey

The survey instrument consisted of demographic questions, free-response questions, and scaled-response (Likert scale) questions. The five demographic questions concerned gender, year in school, major, chemistry courses taken, and EMOC modules used. The four free-response questions asked (a) the advantages/disadvantages of using EMOC, (b) how had EMOC affected their learning, (c) which part of EMOC was most liked and why, and (d) which part of EMOC was least liked and why. The 17 scaled-response statements asking students to rate the statements from AGREE STRONGLY to DISAGREE STRONGLY with a separate category for DID NOT USE. The scaled-response statements were worded both positively and negatively and included oppositely worded paired statements so that we could check the consistency of the students' responses. Table 1 lists the scaled-response statements. This survey was administered to students in the CEM 143 and CEM 251 courses at Michigan State University in the summer of 1998.

Focus Group Protocol

The focus group protocol was designed in seven sections. First, there was a General Background section in which the interviewer introduced him! herself and explained that the purpose of the focus group was to ask students to reflect on the EMOC learning activities and to compare those activities to non-Web-based learning activities. Students were also informed that their participation was voluntary, their responses would be anonymous, they could terminate the focus group at any point, and their participation/non-participation would not affect their grades in chemistry.

Second, we asked general questions about EMOC, such as the parts of EMOC that they used and the general benefits/nonbenefits that they perceived. Third, we asked the groups to comment on the specific issues of flexibility and access, which had emerged from the survey responses. We also asked the students what features of EMOC were important to them, and we asked the students to comment on how they thought that EMOC might have affected their learning.

Fourth, we presented a graph of the effect of pH on acid-base equilibria, which we identified as a topic that was taught in both courses involved in the evaluation (Figure 1). Further, the homework in one course actually had a link to the acid-base tutorial on EMOC. This tutorial allowed students to construct a similar graph as part of the feedback in the tutorial. Therefore, we asked the participants to explain (a) what was happening in the five labeled regions of the graph, (b) what the overall graph meant to them, and (c) the physical significance of the graph. For the web site user focus groups, the graph was introduced as part of the EMOC site. For the nonuser interview, the graph was simply introduced as pertaining to acid-base chemistry.

The graph in Figure 1 represents a system of bromoacetic acid in solution (Equation 1) such that the pH can be externally controlled. The graph, therefore, shows concentrations of bromoacetic acid ("A form," pink line) and bromoacetate ion ("B form," blue line) as a function of pH. With the Acid-Base Tutorial the student can adjust parameters of the system, such as pH and temperature, and observe the effects on concentrations and on the [pK.sub.A] of the acid. From several points within the Acid-Base Equilibria Tutorial, the student can access a help page. This help page provides guidance for student explorations including suggestions to construct graphs of the data collected in the tutorial. These screen instructions are shown in Figure 2.

In the fifth section of the focus group protocol, we asked the participants to construct a concept map of their knowledge of acid-base chemistry. As a training exercise we first showed them a concept map of density using only three terms (density, mass, and volume), and we discussed the basic mechanics of map drawing, such as drawing directional proposition lines, clearly labeling each line and each concept, and drawing as many appropriate links between concepts as possible. Then we showed the participants a more complex map using the terms density, mass, volume, solid, liquid, and gas. We discussed how adding these terms increases the number of relationships possible. Then we presented the participants with a large sheet of paper and Post-It Notes. They were given a list of terms (equilibrium, acid, base, [H.sup.+], [H.sub.3][O.sup.+] conjugate base, water, pH, equilibrium constant, buffer, solution, and [pK.sub.A]) and told that they were to use these terms to construct a concept map that would represent t heir understanding of acid-base chemistry. They were told that they were free to decide not to use terms or to include new terms, as they determined appropriate. We wanted to see if they would, of their own volition, include the term conjugate acid to match the term conjugate base that we had provided. We also wanted to observe what terms they would choose to leave out. They were also instructed to make the map as connected as they thought necessary and to write linking words on all the connecting lines. Further, the connecting lines all needed to have their directionality indicated. After the map was drawn, we discussed the map with the students.

In the sixth section of the focus group protocol, we probed the students as to their preferred mode of taking in information. We gave them a scenario adapted from Nurrenbern (1995), which stated that they were staying in a hotel, had a rental car, and wanted to visit a friend whose address or location they did not know. We then asked them what would be their preferred mode of receiving directions: a map drawn on paper, oral directions, written directions (no map), or just having the general area indicated and letting them find the address by exploration. Finally, we closed the discussion by asking the participants if they had any questions that they would like to ask us about this study. Then we thanked them for their participation.

DATA ANALYSIS

Survey

Demographics and frequency data. Overall, 39 students responded to the survey (20 males, 19 females). The most common majors among the students were packaging (26%), dietetics (10%), and food science (10%). The survey's demographic section asked the students to mark which sections of the EMOC site they had used. Over half of the students (59%) indicated that they had accessed less than three of EMOC's subsections. One-third of the students (33%) indicated that they had accessed 3-4 subsections, and only 8% of the students had accessed more than four subsections.

The three subsections of EMOC that were most heavily used were the Acid-Base Tutorial (59%), the Stereochemistry Tutorial (56%), and the Practice Problems (56%). This confirmed our choice of acid-base chemistry as an appropriate topic for the focus group protocol. One note of interest is that students in one of the organic chemistry classes (CEM 143) were required to use EMOC to submit homework on-line. This homework was a subset of the practice problems offered on the web site, so theoretically, all of the students responding should have used the practice problems.

Scaled response data. The scaled survey responses were tabulated and frequency counts and means were computed for each question. Means for the scaled-response items are given in Table 1. On the scale used, a response of 1 was a positive response to the statement (Strongly Agree), while a 5 was a negative response (Strongly Disagree).

Generally the students' perceptions of EMOC were favorable (Statements 5-15), and they particularly valued the flexibility of the site, as seen from the mean of Statement 7. The Practice Problems and the Acid-Base Equilibrium Tutorial were identified as being the most useful sections of the site. The strongest positive response on the survey was to Statement 7 (mean = 1.71), which probed the issue of flexibility. Only 3 of 34 respondents disagreed with the statement. Because of the strength of this trend, the issue of flexibility was explored in our focus group protocol later in the study. Statement 14 again showed a positive response to the use of EMOC (mean = 3.97) because 24 of 35 respondents disagreed with the statement.

Paired opposite statements were used in the survey in order to check the consistency of the students' responses. If the survey was working properly, a statement with a high positive response should give a high negative response when asked in the opposite manner. Paired opposite statements on the survey suggested that the survey did perform properly. For example, opposite responses to Statements 10 and 8 showed that the ease-of-use/flexibility issue was consistently addressed by the students. Responses to Statements 11 and 15 showed consistency with the added information that 21/35 students (60%) perceived that EMOC helped them learn organic chemistry (Statement 11, mean = 2.43) and that 25/34 (74%) of the students disagreed that EMOC made them more confused (Statement 15, mean = 4.12).

Free response data. The free-response survey results were analyzed by coding the responses using a coding scheme based on the one prepared in early summer using the pilot data from the spring semester. Inter-rater reliability for the data coding was found to be 0.81. This reliability value was calculated by comparing the coding results of two raters. This is considered a good value and is similar to the inter-rater reliability of 0.86 obtained for the pilot survey data collected in Spring 1998.

Across the four questions, the themes that emerged from the free-response data still fell into the main categories of access, features, and learning. The most common category was features, which attracted both positive and negative responses. Students like the flexibility of the site, the homework on the site, the explanations/feedback, the format/layout, the convenience, and the different representations used. However, students expressed concern that sometimes the feedback system just let them hit buttons until they guessed the right answer. Other students sometimes found the format confusing because they did not know how to get the list of homework and due dates from the site.

Under learning, students' comments were mostly positive, and they indicated that the EMOC site did help them learn. They noted that the site 'forced' them to learn and provided more exposure to the material and reinforcement of the concepts. Under the access category, students' comments were again mostly positive in their perception of their access to the site's activities and tutorials and to materials about the course, such as the syllabus and homework due dates. However, they were concerned about access to computers because of the fact that not everyone could get easy access to a computer with the necessary hardware and software to access the Website. These themes were also found in the focus groups, and they are more fully discussed in the following section.

Focus Groups

Focus groups with web site users and nonusers were conducted at Michigan State University. Using the protocol we designed based on the pilot study data, we conducted discussions with three web site users in two separate focus groups and with one web site nonuser in an individual interview. Faculty members for two organic chemistry courses were also interviewed individually.

From the focus group discussions, six strong and two weak/emergent findings were found. Strong findings were those expressed by students in all three parts of the study (scaled-response, free-response, and focus group). Access to materials, to computers, and to places within the site (a concern that we called navigation) was one group of strong findings. Student concern about features such as flexibility, convenience, and site layout were another cluster of strong findings. Finally, a group of strong findings clustered around students' perceptions that they valued the site and that the site had a positive effect on their learning.

Weak or emergent findings were those mentioned in one or two parts of the study. Motivation issues were detected in the focus groups upon transcription and upon closer scrutiny of the free-response survey. Learning style was only addressed in the focus groups.

Strong findings. First, many students positively valued the access that the EMOC site provided them to materials for the course, both in terms of content and administrative matters. In the free-response survey, one student stated: "There is always a second place besides the teacher or book to find out information" (Survey #143-07) This was echoed by a student in a focus group: "She's included everything from the class on the web site....If you don't have the papers available to you, you can just go right to the web site" (female student in User Focus Group #1).

However, the second strong finding showed that access to computers could be a negative issue. Students who did not have ready access to a computer connected to the Internet at all times needed to use a computer at school or at a friend's house. Not surprisingly, this led to negative comments concerning the issue of computer access: "The major disadvantage is that I do not have a computer at home. I do not live on campus so anytime I want to use the computer I have to park and walk, etc." (Survey #143-06). "I don't have a car. I have to take a bus, so that's kind of difficult in itself..." (female student in User Focus Group #1).

Third, students appeared to value the flexibility of the EMOC site in that the program allowed multiple tries to get the right answer. "You get more than one chance to get it right [right arrow] therefore you look up the answer + learn it" (Survey #143-23). "[I]f we don't quite understand the problem, we have as many tries as we want...you can keep working on it until you do understand it" (female student in User Focus Group #1). The opposite side of the issue will appear in the motivation/guessing issue below.

Fourth, students positively valued the convenience of the EMOC site. For many students the value lay in the all-hours availability of the materials, as well as the ability to get needed information via computer: "[S]ometimes I have questions and I can just go on a computer and usually figure out what I'm doing" (Survey #143-12). "Sometimes you're busy all day, and you get to the lab at 10 to 11 at night and try to do the homework then, or you can do it ahead of time..." (Male student in User Focus Group #1).

Fifth, aspects of the format/layout of the site were mentioned as negative issues. The homework assigned through the EMOC site was listed by topic, not by due date. There were problems for some students who did not scroll below to check for homework that was not in the order expected: "The prob[lems] are not in order, it is very easy to miss one b/c you didn't see it" (Survey #143-23). "They're not chronological, so...if you go through five in a row, and then it has one due for different days, you know, that's it for today. And you missed a few problems that you didn't see because you didn't check..." (male student in User Focus Group #1).

Sixth, students generally believed that the EMOC site helped them learn the content. Some responses mentioned that the constant homework due dates (which would be impractical to use without web-based homework) made them learn. Others viewed the matter more holistically: "It has forced me to utilize all that is available to me" (Survey #143-04). "I went from a zero-point to a four-point. You can't dispute that" (male student in User Focus Group #2).

Perhaps the clearest insight was from a student who commented that "[EMOC] has affected how I learned organic chem. It has affected my learning because I feel as if I am playing a role in my education. Unlike the traditional method of lecture and reading followed up by regurgitation on test[s], I played a more interactive role in my education" (Survey #143-03).

Weak/emergent findings. First, motivation issues were uncovered in the analysis of the focus group discussions. Because the web-based activities in EMOC are individual and for the most part optional, the student's motivation to use the materials makes a difference in if and how the materials are used. Even for the required homework problems, motivation issues (in the form of guessing) emerged in the process of data analysis. Because unlimited tries were allowed (see the third strong finding above), some students noted that you could guess without learning anything: "For the non-dedicated student, they can be easy bonus points because you can guess on the answers" (Survey #143-15). "The way the homework is set up, you can kinda just guess x many times and get the right answer....I guess pretty much I wasn't motivated" (female student in Nonuser Focus Group).

General issues of motivation also surfaced concerning the content matter. For this student, it wasn't that the web site was not useful; it was a matter of not being interested in organic chemistry: "I think that if a student was more motivated to learn chemistry, it would be a great tool to have, because you can learn so much....For the majority of students, learning organic chemistry is not a major priority in our lives" (female student in Nonuser Focus Group).

Another issue of motivation was described by a student who had difficulty making a required plug-in work in a public campus computer lab: "I just went up to my professor's--up in her office and she'd get it up on her computer and show me. I mean, it's an extra step and it's not like I wasn't willing to take that extra step, but somebody might not be... The whole professor/student interaction sometimes can be intimidating" (male student in User Focus Group #2). For a student to take care of an access problem in such a manner requires the motivation to approach the professor and describe the problem, and ask the professor to take action.

Analysis of graph explanations. Students demonstrated in the focus groups that they could generally explain the concentration trends sufficiently, but they had difficulty in explaining region E. A student in User Focus Group 1 stated that region E was where the solution was at equilibrium. This idea was also expressed by the student in User Focus Group 2 and by the student in the nonuser focus group. In the words of the nonuser: "Where it says E, that's the equilibrium...at this point you'd be less likely to have acids donating and bases accepting because it's at equilibrium." Converse to the students' indication that point E is the location of equilibrium, the system represented on the graph can actually be at equilibrium at any pH value. The ratio of acid to base can change, but equilibrium is maintained at any pair of concentrations on the lines at a given pH.

Another interesting interpretation of the graph came from the student in User Focus Group 2, who interpreted the graph as representing a reaction of an acid and an externally added base, such as the situation for an acid-base titration. This EMOC user discussed areas C and D on the graph:

Student. It looks like in that reaction or whatever, the acid's all used up. And there's nothing left.... There's going to be no more acid. It'll turn into salt. Sodium chloride.... And then as the A line comes down and the B line is coming up and the A line disappears into the D region, all that's left is the line in the C region. That means that all I've got there is base. The reaction completely--there's nothing left except the base.

Interviewer. Okay. So there's nothing left, and that's over by C and D?

Student. Yeah. Like it was used up and it--'cause a base and an acid's gonna get together; you get a lot of base, it's gonna tear apart the acid. There's gonna be no more acid. It'll turn into salt. Sodium chloride. Interviewer. So is that a kind of physical representation that you think of when you see this [graph] is something like HCI and sodium hydroxide?

Student. Right. Right. It would be an acid that would reach, you know, a reaction here.

From the student's statements, it appears that the student is treating the graph as a representation of a reaction between a strong acid and a strong base, such as a general chemistry titration of hydrochloric acid with sodium hydroxide, rather than a representation of a weak acid/conjugate base system such as bromoacetic acid and bromoacetate ion.

The differences between the users and nonuser in the explanations of the graphs came largely from the nonuser's discussion of protons as they relate to acids and bases. The nonuser explained that bases are proton acceptors and acids are proton donors, while students in User Group 1 simply mentioned protons but did not elaborate. The users of EMOC talked more about the graph itself rather than the background of the graph.

All of the focus group participants were asked to describe the physical situation represented by the graph. One web site user drew a description, but her description on paper was a representation of a macroscopic transfer of water between tanks that she had seen as a demonstration of equilibrium. None of the students in the focus groups could or would draw a molecular-level representation of what the graph represented. While most of the statements in the discussions were correct, the majority of the statements pertained to technical aspects of the graph itself rather than physical significance of the graph.

Possible improvements to this area of the Acid-Base Tutorial could include better distinction of this equilibrium system as being a weak acid/conjugate base system, as opposed to a representation of the reaction between a strong acid and a strong base. Also, links from various portions of the graph to separate explanatory pages could be most helpful. For example, clicking on the graph in the area around A could lead to a short discussion of the chemistry that happens there. Each of the individual explanations could remind the student that the equilibrium process between the weak acid and its conjugate base is always happening, but at different positions.

Analysis of concept maps. Students constructed concept maps concerning acid-base chemistry after the graph interpretation part of the focus groups. Students were briefed on the design of concept maps and shown examples of two maps of the concept of density before being given a list of concepts to use in their maps. Figure 3 contains a sample concept map. Students were allowed to add concepts or exclude concepts as they saw fit. The student concept maps were analyzed by determining the correctness of the links, the terms added and/or not used, and the number of links entering and exiting each concept. Counts of the terms used in the student concept maps and lists of terms not used and added are shown in Table 2.

It can be seen that the EMOC nonuser added more concepts to the map than were supplied, while only one EMOC user (in Group 2) added any concepts. The links in the student concept maps were analyzed for correctness, and links were categorized as correct, incorrect, or incomplete. A summary of the linkages made by the students appears in Table 3.

We began to speculate that perhaps the female nonuser student had not used the site because she felt that she already understood some of the material. This speculation was checked as we analyzed the rest of the focus group discussions.

It can be seen that the nonuser had the most correct links between concepts, while the users of EMOC did not make as many links and had more incorrect or incomplete links. In all of the student concept maps, the concept of equilibrium is deficient. Only one correct link to or from equilibrium was made in all three concept maps. One concept map did not even include equilibrium as a concept. This map did include equilibrium constant, but only in the link "p[K.sub.A] [is] -log of equilibrium constant." Although the nonuser identified more correct linkages, the student did not include the concepts of equilibrium or buffer, which the users groups did attempt to use.

Again, we speculated that the female nonuser had a better command of the topic and therefore had little motivation to actively use EMOC. We suspect that EMOC was seen as a valuable resource by the less well-prepared students, and we speculate that EMOC's contribution to changes in students' understanding will be made in moving lower achieving students towards a greater understanding.

To gain a better picture of how the student concept maps compared, the same expert who explained the graph earlier was requested to draw a concept map using the terms supplied to the students. Not surprisingly, the expert map was quite a bit more extensive in its linkages. For comparison, the numbers of links entering and exiting each concept of the students' maps were computed.

The total numbers of incoming and outgoing links for the expert concept map were computed for each of the concepts used in the map. From the totals of incoming and outgoing linkages made in the expert's concept map, we identified the concepts with the highest totals of both types of linkages. The concepts acid, solution, water, pH, and base were identified to have the large numbers of links incoming. We argue that a concept with a large number of incoming links is one that is important to a number of other concepts. Other concepts depend somehow on the concept with incoming links. Also, these concepts had large numbers of links outgoing, which means that the concepts in question can be related to other concepts. Based on the analysis of the expert's concept map, concepts with 14 or more links were determined to be key concepts.

The student maps' totals of incoming and outgoing links for the concepts identified as key to the expert map were compared to the expert's totals. The students' total numbers of incoming and outgoing links were calculated for the expert's five key concepts, then links that were incorrect or incomplete were subtracted. After adjusting for the correctness of the students' links, user group 1 had 3 correct links into and 2 correct links out of the expert's key ideas. User 2 had 1 and 3 links in and out, respectively, and the nonuser had 3 links into and 8 links out of the expert's key ideas. The nonuser's map would be considered closest to the expert's concept map in terms of linkages to the key ideas of the expert. User Focus Group 1's concept map was next closest, followed by User Focus Group 2.

The interview with the EMOC nonuser revealed why the student did not use the web site. When asked why she did not use the web site, the nonuser replied, "Basically because I didn't feel I needed to....I guess I pretty much wasn't motivated." This student may have felt that she did not need to use the tutorial materials on the EMOC site because she already had sufficient mastery of the material. This mastery was demonstrated in her explanation of the graph in terms of acceptance of protons rather than simply explaining the graph in terms of concentration trends of acid and base, which all of the students interviewed could do. She also demonstrated her mastery of the material in her concept map, which had no incorrect links and only one incomplete link. The mastery of the material exhibited is consistent with the student's statement that she did not need to use the web site.

SUMMARY AND CONCLUSIONS

Guiding Questions

Our evaluation was guided by three questions. The first question probed what students perceived as the advantages and disadvantages of EMOC. The evaluation showed that students perceived as advantages the access to materials afforded by the EMOC site (particularly the on-line homework), the feedback system, the flexibility of the site, and the convenience of having the site available. Disadvantages cited by students dealt with issues of access to computers and appropriate software and the sometimes confusing format/layout of the site.

The second question asked for the parts of the site that students liked most or liked least. As can be seen from Table 1, students were generally favorable to all parts of the site.

The third question addressed the impact of EMOC on students' understanding of organic chemistry. In the surveys and focus groups, most students indicated that they believed that EMOC had clarified their understanding of the content matter. However, the students' explanations of the acid-base graph and their concept maps of acid-base chemistry showed that the nonuser performed better that the users. On the other hand, one user was adamant that EMOC had helped him move to a much higher grade in the course than he had earned previously. Based on these considerations, we speculate that EMOC is of most direct benefit to students' with weaker backgrounds. This may be strongly influenced by motivation because students with weaker backgrounds may be more motivated to seek out this extra help. Discussions with larger groups of users and nonusers could resolve this issue.

Improvements Needed

Students stated concern about computer access off campus and after hours. While this is not necessarily an issue that the developer of EMOC could be able to handle alone, it is an important one for developers of other web-based materials. Also, students felt that format changes to clarify the due dates of assignments would be helpful. We found that the students interviewed had difficulty in realizing that the graph of pH effect on Acid-Base Equilibria dealt with an acid and its conjugate base, rather than a reaction (such as a titration) between two separate species. Given the importance of pH on acid-conjugate base equilibria for organic chemistry, it would be beneficial for improvements to be made in this area.

Acknowledgements

The authors gratefully acknowledge the support provided by National Science Foundation grant HRD-9752985 for this project and its evaluation. A.P. also gratefully acknowledges Dr. Gerd Kortemeyer for the development and maintenance of the LectureOnline system in which the graded homework problems were developed and monitored. The authors also wish to thank the students and faculty of the organic chemistry courses at Michigan State University who cheerfully cooperated with us in the collection of this data. We also thank Deanna Auberry and Eric Malina for performing the scoring needed to calculate the inter-rater reliabilities cited in this study and Dr. John Nash for his willingness to construct an expert concept map.

References

Conklin, J. (1987). Hypertext: An introduction and survey. IEEE Computer, 2(9), 17-41.

Halasz, F., & Conklin, J. (1989). Issues in the design and application of hypermedia systems. Austin: CHI '89 Tutorial #26.

Heller, R.S. (1990). The role of hypermedia in education: A look at the research issues. Journal of Research on Computing in Education, 22, 431-441.

Jones, W. (1987). How do we distinguish the hyper from the hype in nonlinear text? In H. Buillinger & B. Shackel (Eds.), Human-computer interaction--Interact '87. New York: North-Holland.

Kortmeyer, G., & Bauer, W. (in press). Multimedia collaboative content creation ([mc.sup.3]), the MSU LectureOnline system. Journal of Engineering Education.

Kozma, R.B. (1991). Learning with media. Review of Educational Research, 62, 179-211.

Lawless, K.A., & Kulikowich, J.M. (1996). Understanding hypertext navigation through cluster analysis. Journal of Educational Computing Research, 14, 385-399.

Markham, K.M., Mintzes, J.J., & Jones, M.G. (1994). The concept map as a research and evaluation tool: Further evidence of validity. Journal of Research in Science Teaching, 31(1), 91-101.

Melara, G.E. (1996). Investigating learning styles on different hypertext environments: Hierarchical-like and network-like structures. Journal of Educational Computing Research, 14, 313-28.

Nakhleh, M.B., & Donovan, W.J. (1998). [Expert concept map of acid-base concepts from the EMOC web site]. Unpublished data.

Novak, J.D. (1976). Understanding the learning process and effectiveness of teaching methods in the classroom, laboratory, and field. Science Education, 60(4), 493-512.

Nurrenbern, S.C. (Ed.). (1995). Experiences in cooperative learning: A collection for chemistry teachers. Madison, WI: Institute for Chemical Education, University of Wisconsin-Madison.

Parrill, A.L. Educational materials for organic chemistry (EMOC). Available URL: http://www.cem.msu.edu/[sim]parrill/.

Raskin, J. (1987). The hype in hypertext: A critique. Chapel Hill, NC: Proceedings of Hypertext '87, 76-87.

Singer, R. (1977). To err or not to err: A question for the instruction of psychomotor skills. Review of Educational Research, 47 (3), 479-498.

Stewart, J., Van Kirk, J., & Rowell, R. (1979). Concept maps: A tool for use in biology teaching. The American Biology Teacher, 41(3), 171-175.

Wallace, J.D., & Mintzes, J.J. (1990). The concept map as a research tool: Exploring conceptual change in biology. Journal of Research in Science Teaching, 27(10), 1033-1052.

Wandersee, J.H. (1990). Concept mapping and the cartography of cognition. Journal of Research in Science Teaching, 27(10), 923-936.

Windschitl, M. (1998). The WWW and classroom research: What path should we take? Educational Researcher, 27(1), 28-33.

                    Means of Scaled Response Statements
Scaled Response Statement                                     Mean [*]
 5. EMOC provided a new way to learn organic chemistry.       2.03
 6. EMOC allowed me to learn concepts that would
    be difficult to learn otherwise.                          2.46
 7. I liked using EMOC because I could work at my own pace.   1.71
 8. Navigating in EMOC was confusing for me.                  3.83
 9. Using EMOC made me more confident about learning
    organic chemistry.                                        2.77
10. Using EMOC was easy.                                      2.06
11. EMOC helped me to understand the concepts of
    organic chemistry.                                        2.43
12. I enjoyed using the EMOC materials.                       2.54
13. EMOC motivated me to learn more about organic chemistry.  2.86
14. I would rather NOT use EMOC to learn organic chemistry.   3.97
15. Using EMOC made me more confused about organic chemistry. 4.12
16. The Organic Reaction Roadmap was useful to me.            3.00
17. The Acid-Base Equilibria Tutorial was useful to me.       2.48
18. The Stereochemistry Tutorial was useful to me.            2.53
19. The Practice Problems were useful to me.                  2.33
20. The Cycloalkanes Page was useful to me.                   3.00
21. The Carbohydrate Structure Activities were useful to me.  2.86
(*.)(1 = Strongly Agree; 5 = Strongly Disagree)
                         Concept Usage in Student-
                             Constructed Maps
        Used                 Not Used             Added          Correct
User 1  equilibrium          [H.sup.+]                              2
        acid                 [H.sub.3][O.sup.+]
        base                 conjugate base
        pH                   water
        buffer               equilibrium constant
        solution
        p[K.sub.a]
User 2  equilibrium          [H.sup.+]            reaction          4
        acid                 [H.sub.3][O.sup.+]
        base                 water
        conjugate base       solution
        pH
        equilibrium constant
        buffer
        p[K.sub.a]
Nonuser acid                 equilibrium          p[K.sub.b]       13
        base                 pH                   conjugate acid
        [H.sup.+]            buffer               strong acid
        [H.sub.3][O.sup.+]   solution             weak acid
        conjugate base
        water
        equilibrium constant
        p[K.sub.a]
        LINKS
        Incorrect Incomplete
User 1      4         2
User 2      3         1
Nonuser     0         1
                          Links Between Concepts
                              in Student Maps
Focus Group Correct Pairs
User 1      acid[right arrow]solution
            base[right arrow]solution
User 2      acid[right arrow]reaction
            base[right arrow]reaction
            equilibrium[right arrow]conjugate base
            p[K.sub.a[right arrow]reaction
Nonuser     water[right arrow][H.sup.+][R][H.sub.3]
            [O.sup.+]
            base[right arrow]p[K.sub.b]
            base[right arrow]acid
            acid[right arrow]base
            acid[right arrow][H.sup.+]
            acid[right arrow]conjugate base
            acid[right arrow]p[K.sub.a]
            p[K.sub.a][right arrow]strong acid
            p[K.sub.a][right arrow]weak acid
            p[K.sub.a][right arrow]equilibrium
            constant
            base[right arrow]conjugate acid
            [H.sub.3][O.sup.+][right arrow]conjugate
            acid
Focus Group Incorrect Pairs
User 1      solution[right arrow]pH
            solution[right arrow]p[K.sub.a]
            pH[right arrow]p[K.sub.a]
            equilibrium[right arrow]buffer
User 2      pH[right arrow]equilibrium constant
            reaction[right arrow]equilibrium
            base[right arrow]buffer
Nonuser
Focus Group Incomplete Pairs
User 1      pH[right arrow]equilibrium
            p[K.sub.a][right arrow]equilibrium
User 2      equilibruim constant[right arrow]p[K.sub.a]
Nonuser
            water[right arrow]base