Assessing complex problem-solving skills and knowledge assembly using web-based hypermedia design.

This research project studied the effects of hierarchical versus heterarchical hypermedia structures of web-based case representations on complex problem-solving skills and knowledge assembly in problem-centered learning environments to develop a system or model that informs the design of web-based cases for ill-structured problems across multiple disciplines. Two groups of students were assigned to work through an ill-structured problem, represented hierarchically and heterarchically in a web-based format. A web-based tracking program was deployed to track students' journeys through the hypermedia case designs. Students were observed while interacting with the problem and were interviewed after submitting their case solutions. Results from the tracking program, observations, case solutions, and interview questions will address case design issues, problem-solving issues, and group processes.

RESEARCH PROBLEM

Several research projects and studies (Booth-Sweeney, 2001; Herreid & Schiller, 2001; Siegel, et al., 2000; Rogers & Erickson, 1998; Gerdes, 1998; Sutyak, Lebeau, & O'Donnell, 1998; Fitzgerald & Semrau, 1996; Jacobson, Maouri, Mishra, & Kolar, 1996) have investigated the use of cases or problems in instruction particularly in relation to case structure (e.g., linear vs. hypertext, narrative vs. conceptual) and problem complexity (e.g., well-defined vs. ill-structured), and the impact of such variables on advanced knowledge acquisition. However to date, there is no explicit framework or instructional model to guide the design of web-based hypermedia cases, particularly for ill-structured problems or learning tasks that require students to engage in complex problem-solving and knowledge transfer. In addition, Grissom and Koschmann (1995) contended that cases or problems that are used as a stimulus for authentic activity are hard to come by in disciplines other than medicine, business, and law, and suggested tha t hypertext/hypermedia may be a more efficient and effective medium to produce cases for learning environments that are problem-centered.

This research project studied the effects of different hypermedia structures of web-based case representations on complex problem-solving skills and knowledge assembly in problem-centered learning environments to develop a system or model that informs the design of web-based cases for ill-structured problems across multiple disciplines. Ill-structured problems are the kinds of problems or tasks that are encountered in everyday practice requiring the integration of several content domains and possessing multiple solutions or solution paths (Jonassen, 1997). They are problems that are "situated in and emergent from a specific context" (Jonassen, 1997, p. 68) and "lacking solutions that are indisputably correct" (Kagan, 1993, p. 715). Therefore they are arguably most appropriate for engaging students in authentic activity and assessing complex problem-solving skills and knowledge transfer.

THEORETICAL FRAMEWORK

Duffy and Cunningham (1996) identified five strategies for using problems in instruction: (a) the problem as an example; (b) the problem as an integrator or test; (c) the problem as a guide; (d) the problem as a vehicle for process; and (e) the problem as a stimulus for authentic activity. This research project focused on their strategy of using the problem as a stimulus for authentic activity. Authentic activity is most simply defined as the ordinary practices of a culture, namely, coherent, meaningful, and purposeful activities (Brown, Collins, & Duguid, 1989). More specifically, an authentic learning task for students should have the following characteristics. It should: (a) cue the leaner to the desired solution(s) in order to promote "free exploration" or self-directed inquiry; (b) allow multiple solutions or solution paths; (c) have no explicit means for determining appropriate action; (d) be perceived as real or consequential by the learner to promote ownership of the problem; and (e) possess multiple criteria for evaluating solutions.

Problems or learning tasks that possess the previously described characteristics are known as ill-defined or ill-structured problems (Jonassen, 1997, 2000). In mathematics education, these learning tasks are sometimes referred to as "thought-revealing" or "model-eliciting" problems (Lesh, Hoover, Hole, Kelly, & Post, 2000). In this genre of problems, the onus is placed on the student to create a model to capture in meaningful ways, the complexity presented in the learning task. The central question for the task designer becomes, "Does the task put students in a situation where they recognize the need to develop a model for interpreting the givens, goals, and possible solution processes in a complex, problem-solving situation? Or, does it ask them to produce only an answer to a question that was formulated by others?" (Lesh, et al., 2000, p. 609).

Problem-based learning (PBL) is one of the most extensive and elaborate instructional models in designing authentic learning tasks (Dabbagh, Jonassen, Yueh, & Samouilova, 2000). PBL is defined as "the learning that results from the process of working toward the understanding or resolution of a problem" (Barrows & Tamblyn, 1980, p. 18). The learner is expected to actively participate in the learning process and construct his/her own knowledge through direct experience and interaction with the situation, that is, knowing-in-action (Schon, 1987). As Stepien and Gallagher (1993) stated: "Problem-based learning is apprenticeship for real-life problem-solving" (p. 26). PBL problems are typically ill-structured and complex, revealing the underlying principles and concepts of a knowledge domain through descriptions of real life events and experiences, and serving as the stimulus and focus for student activity (Siegel et al., 2000; Norman & Schmidt, 1992; Stepien & Gallagher, 1993). PBL radically alters the role of th e teacher from instructor to facilitator and as time goes on, learners become increasingly self-directed in the learning process (Shoffner & Dalton, 1997) by taking ownership of the problem and "assuming the roles of scientists, historians, doctors, or others who have a real stake in the proposed problem" (Stepien & Gallagher, 1993).

HYPERMEDIA SUPPORT FOR PBL

The use of hypertext/hypermedia to represent PBL cases has been a primary interest to researchers of PBL due to its inherent nonlinear interlinking ability in representing a complex and interconnected body of information (Hemstreet, 1997; Jacobson & Spiro, 1995). According to Hemstreet (1997), hypermedia can support the complexity of PBL environments by situating a complex problem in a story format and making additional information about the problem readily available, similar to how information is built into anchored instructional materials. Hemstreet added that hypermedia can also provide a database of information where more information about the problem than necessary is included, and the organization of the information provides no hints about what is important or relevant to the problem. These hypermedia problem-representation capabilities support the characteristics of ill-structured problems and could provide an efficient and effective medium to design cases for learning environments that are problem-cen tered.

Additionally, hypermedia cases provide students with the capability of browsing the features of a case in a nonlinear, multidimensional, semantic structure that mimics the brain's ability to store and retrieve information by referential links for quick and intuitive access (Keep & McLaughlin, 1995). This information seeking activity is designed to support an authentic process of inquiry consistent with problem-based pedagogy. It is a non-sequential activity that can be facilitated in a deliberate way by arranging information in a format within which exploration can take place. Developers, be they teachers or instructional designers, must chart out the knowledge students will "run into" in the course of working on a problem or case, and compare this to the stated goals of a course (Stepien & Gallagher, 1993). The question is however, how does one determine the degree and type of structure with which to arrange case information, and does it matter?

Sutyak et al. (1998) argued that case structure may have a profound impact on knowledge transfer and students' expectations of learning, and that case structure remains a fertile field of study. Additionally, Siegel et al. (2000) suggested that a high level of transfer might be attained through web-based instructional design that supports case- and problem-based instruction and learning. Siegel et al. cautioned however that despite the powerful hypertext capabilities of the World Wide Web (WWW or Web), many casebased designs continue to reflect a "traditional hierarchical approach, modified by extended page-linking capabilities" (p. 2), inhibiting advanced knowledge acquisition and transfer.

CASE STRUCTURE IN WEB-BASED HYPERMEDIA LEARNING SYSTEMS

To test the effects of case structure on problem-solving and knowledge transfer, this research project focused on hierarchical versus heterarchical web-based hypermedia designs of ill-structured problems. Hierarchical hypermedia designs are linking structures (structures that facilitate navigation through a hypermedia system) that organize content into logical sections often by major topic area, resulting in several navigation levels with a top-down or tree-like structure (Oliver, 1996; Last, O'Donnell, & Kelly, 2001). Network-like or heterarchical structures are "more chaotic with random links jumping from one topic to another in a more referential fashion" (Oliver, 1996, p. 15). Typically, embedded links are sprawled throughout the content representing associations similar to knowledge representation in a memory model or network structure (i.e., rhizome-like). Hierarchical versus heterarchical hypermedia designs can also be viewed as a "depth versus breadth" topology of linking structures (Larson & Czerwins ki, 1998). For example, in a hierarchical or tree-like hypermedia structure, the deeper a student browses the more detail about the topic is encountered, and to change themes (topics) the student must return to the highest level (Last et. al, 2001).

Although research indicates that a more tree-like or hierarchical structure limits navigational difficulties when browsing through a hypermedia learning system compared to a purely heterarchical structure, there is evidence that linking structures interact with learner characteristics (e.g., learners' prior knowledge and cognitive styles) and with the goal of the learning task (e.g., studying for an exam vs. exploring or searching for specific information), and that the design characteristics of a hypermedia learning environment can enhance or limit user performance and impact the quality of independent learning (Last et. al, 2001). To further explore the interaction between hypermedia linking structures and student learning and performance, particularly in a web-based hypermedia learning environment, this research project addressed the following two research hypotheses:

1. In a heterarchical (rhizomatic, network-like) hypermedia design of an ill-structured problem, significant improvement in problem-solving schemes would be observed in students who regard learning as an active process of constructing meaning.

2. Students engaged in solving an ill-structured problem of heterarchical hypermedia design will demonstrate improved knowledge transfer compared to students engaged in solving an ill-structured problem of hierarchical (tree-like, topically organized) hypermedia design.

METHOD OF INVESTIGATION

To empirically address the research hypotheses, the first phase of this project is an exploratory case study to establish a set of assumptions that will guide future phases (Robson, 1993). This first phase serves as a diagnostic tool to study the nature of the independent variable (case structure) and the learning interactions, strategies, and exploratory pathways it generates in order to design an effective experimental study in future phases. Students' beliefs and perceptions about case structure were examined by engaging them in a web-based, PBL environment to solve an ill-structured problem of hierarchical and heterarchical hypermedia design. The following research questions were addressed in this study:

1. How does exploration in the heterarchical hypermedia case compare to exploration in the hierarchical hypermedia case? Is one more comprehensive than the other? How many links did students visit? Were the links visited relevant or irrelevant to the case? How much additional information was explored? How much time was spent on a page/link? How many links were revisited to gain more clarification of the case issues?

2. How does collaboration between group members in the heterarchical hypermedia case compare to collaboration in the hierarchical hypermedia case? Were students' decisions in terms of how to navigate the case content group-based or individually based? Were students' decisions on what actions to take while journeying through the hypermedia learning environment group-based or individually based?

3. How did students in each of the groups perceive the learning task in each of these case designs? Was the learning task perceived as authentic, challenging, complex, ill-structured, meaningful, relevant, and engaging? Were these perceptions different in each group?

4. How did students perceive the case structure in each of the groups? Did they find it difficult to navigate through the case? Why or why not? Did they find it difficult to locate specific information?

The ill-structured problem selected for this study is an Instructional Design (ID) case concerning the issue of "informed consent" prior to surgery. ID cases are archetypal examples of ill-structured problems (Jonassen, 1997) and have been selected as study materials due to the expertise of the researcher in this knowledge domain. The ID problem used in this research study relates to preoperative instruction in which potential gall bladder disease patients learn enough about the surgical procedure to give "informed" consent. The students are responsible for designing instruction and assessment that legally verifies that patients were fully informed. If such instruction could be validated as effective, huge benefits could accrue to the medical profession and certainly to the insurance companies given the impending number of malpractice cases filed against physicians for either misinforming or not informing patients about possible risks of gall bladder surgery and/or alternative procedures to correct the diseas e.

This informed consent ID case was used in a previous research study (Dabbagh, et al., 2000) to argue that ID is a dynamic process of problem understanding and problem solution and that problem-based learning is an effective instructional method to teach ID because of its strategy for using problems as a stimulus for authentic activity and its emphasis on preparing learners to become lifelong critical thinkers. The case material was initially compiled in print form and presented to student groups in the Dabbagh et al. case study in a folder labeled by Barrows (1985) as the problem-based learning module (PBLM). The PBLM contained the problem statement, authentic malpractice court cases, samples of patients' medical histories, articles related to the problem of informed consent, samples of legal informed consent forms, and legal definitions of informed consent from Black's Law Dictionary. Students were also provided with related videos that demonstrated physicians explaining gall bladder disease and its treatmen t options and possible complications to patients.

The informed consent ID case has since been converted to a web-based hypermedia design reflecting the two types of hypermedia linking structures previously discussed: hierarchical and heterarchical. The hierarchical structure consists of an unmistakable and unique entry point to the case which begins with a narrative of the problem scenario and branches to a web page with a menu of hyperlinks to six topically labeled case themes: legal cases, definitions, article summaries, legal forms, patient information, and perspectives of stakeholders in the informed consent issue. Each of these themes is further subdivided into sub-topics menus of hyperlinks reflecting a tree-like navigational structure requiring students to browse deeper into the linking structure for more case detail, and to return to higher levels in the hierarchical organization to change themes and sub-topics. Navigation in the heterarchical structure on the other hand is enabled through embedded textual hyperlinks that embody the natural meaning o f the narrative articulating the case information and relevant components. The hyperlinks are sprawled randomly across the case content representing referential associations that resemble a network-like or rhizomatic structure with no discernable hierarchy or repeatable paths. Furthermore, the heterarchical hypermedia structure describes the case issues and events in a story-like manner using real world simulations of characters and role-plays deepening perhaps the authenticity of the problem context. The story-like nature of the heterarchical linking structure will be considered an attribute of the heterarchical design and its implications on student perceptions of the learning task will be considered only to the extent to which students perceive it as an important or distinctive variable of the hypermedia learning environment.

DESCRIPTION OF STUDY

Two groups of volunteer students in their last semester of study in a graduate program in ID at a large comprehensive university were assigned to explore the two case designs described. The researcher initially sent an email request to 15 students asking them to volunteer to participate in the study. Six students responded that they would be willing to participate (40% response rate). The first three who responded were assigned to the heterarchical case design and the last three were assigned to the hierarchical case design. Students were told that the study will take about three hours of their time and that they would each be given a $20 gift certificate from a major bookstore in the area to compensate for their time and effort. Each group of students was asked to come to a computer lab on a specific date and the students were given instructions on how to access the case. The group assigned to the heterarchical case design was identified as the red group and the group assigned to the hierarchical case was i dentified as the purple group. Students in each group were instructed to use only one computer and select an operator amongst them (person sitting at the keyboard) who will be in charge of accessing the case online and clicking the mouse. The procedure for accessing the web-based case required students to enter their group name, their individual names, and the operator's name. Then each group entered the URL corresponding to their case design assignment and the following instructions were displayed:

This is an Instructional Design case study. Study the case and provide a one-page outline of your recommended solution in a Word document. While studying the case, you can highlight text, copy and paste portions of text, click images or links, print, etc. Every time you take an action, you will be asked why in a pop-up menu. Please be patient and answer as candidly as possible. You will not be judged or evaluated in anyway. This is a research experiment. When done, you will be asked to respond to a questionnaire to give us your suggestions. Thank you for participating in this study. Click on the arrow to enter the case.

Each student group was observed by the researcher while interacting with the case and their journeys through the hypermedia structure were tracked to a database (through a JAVA program) using a pop-up menu that automatically displayed when students used any navigation tool to move through the pages and links of the case and/or interact with the case content (e.g., clicked on a hyperlink, highlighted text, copied or pasted text, printed text, used browser navigation tools to move backward and forward, etc.). The pop-up menu requested short answers to embedded elaboration-seeking questions and required students to make selections from a list of choices to help determine: (a) whose decision was it to initiate the interaction with case content (group versus individual), and (b) why was that interaction selected (relevancy to problem-solving process). There were five options to choose from in the second question: (a) very helpful for case analysis; (b) somewhat helpful; (c) not sure, but decided to explore anyway ; (d) didn't have any other option; (e) other reason (Figure 1).

A slightly different pop-up menu containing the options for question 2 was displayed when students exited the page that was visited, or completed the action that was selected, to determine whether students' initial perception about the usefulness or relevancy of the link/action to problem-solving was confirmed. This pre/post tracking technique was necessary to determine what students perceived as relevant and irrelevant case information (Figure 2).

In addition to the information collected in Figures 1 and 2, the JAVA program recorded the URL of every page visited, the URL of every page in which an interaction was initiated, the start time of an interaction, the type of interaction, the time spent on every page/interaction, and the total session time. During each group session the researcher was silently observing the group processes and deliberations triggered while exploring the case content and taking notes. At the end of the session, each group submitted their outline of the case issues and recommended solution, and the researcher proceeded to interview each group and explained that during the interview group members may jump in and answer questions at any time. The interviews were audio recorded and later transcribed for further analyses. The interview questions were as follows:

1. What did you think of the case in general? Was it easy to understand? Complex? Ill-defined? Well-defined?

2. What did you think of the topic of informed consent? Was it realistic? Relevant? Meaningful? Was it a timely topic? Did it engage you in the case?

3. What did you think of the structure of the case? The way it was laid out? The navigation? The organization of the links and resources?

4. Was it easy to find what you were looking for?

5. Describe the general strategy you adopted when using the system or navigating through the case. In other words, did you have a plan of attack?

6. Do you think you had enough information to identify the main issues! problems to solve the case?

7. Do you think all of the information presented in this case was useful?

8. How would you change the case structure or present it differently?

ANALYSIS AND DISCUSSION

This research investigation used a case study model. In addition to being an exploratory case study, this first phase is what Yin (1993, 1994) referred to as a positivist case study model where the researcher is attempting to establish the facts of the case by gathering evidence from a variety of resources and applying the concept of triangulation, which treats the evidence in a converging manner making any theory building inferences robust and viable. A qualitative approach was used to analyze the data. The primary data analysis techniques included content analyses of the interview reports and descriptive statistics of the data collected from tracking students' journeys through the two web-based hypermedia case designs. These two different analyses techniques support the concept of methodological triangulation in case study research (Denzin, 1978; Patton, 1990). The researcher's recorded observations and the groups' one-page outlines of the case issues and recommended solutions were used as data/source tria ngulation (Denzin, 1978; Patton, 1990) to cross check the accuracy of the data collected. Methodological and data/source triangulation ensure the credibility of this exploratory case study. To carefully organize and interpret the data using the described techniques, a data analysis matrix (Table 1) was developed that maps the four research questions of this case study to corresponding components of the primary data sources. Table 2 shows the descriptive statistics and findings from the JAVA tracking program, and Table 3 shows the results of the interview that was conducted at the end of the study. A discussion of the data analyses and implications for each research question is presented.

Research question 1: How does exploration in the heterarchical hypermedia case compare to exploration in the hierarchical hypermedia case? Is one more comprehensive than the other?

In terms of the breadth of exploration in the hypermedia cases the results showed that the purple group's journeys (hierarchical) were more comprehensive overall. The purple group visited 54% of the links compared to 41% for the red group (heterarchical). The number of links revisited by the purple group was also higher than those revisited by the red group (39% compared to 33%). Additionally, the number of unique links visited by the purple group was higher than the number of unique links visited by the red group (32% compared to 27%) (Table 1). However, the total time spent interacting with the case content was higher for the red group (2 hours and 40 minutes compared to 2 hours and 7 minutes). The red group spent an average of 3.04 minutes on a page compared to 1.96 minutes for the purple group. This indicates that the heterarchical case structure may have caused the participants to reflect more meaningfully on the case content, studying and discussing the information more closely to formulate a problem so lution that was appropriate. The data from the interview supports this finding. In response to interview question 6, the participants in the red group indicated that they "got immersed in the material" and had to "stop and look at what we had, and go step by step through what we learned and what we gathered to be able to reframe the issue.

On the issue of usefulness and relevancy of case information to problem-solving, the participants in the red group indicated that some of the information was not useful but that "only after looking at the information we were able to determine that it was not useful" and that "even the unuseful information was useful in helping us come up with a conclusion." The participants in the purple group pointed out that all the information was useful and that it was "good" information but did not elaborate on what information may have been more useful in helping them understand the issues of the case. The interview analyses indicated that the purple group seemed more focused on getting to information than on debating the issues and perspectives of the case content. The descriptive data indicated that both groups had the same percentage of links (5 5%) initially perceived, and later confirmed, useful, for understanding the case. However the red group had a higher percentage of links, 31%, initially perceived useful but later confirmed not useful, compared to 27% in the purple group. Additionally, the descriptive data indicated that the frequency reported for reason (c), "not sure but decided to explore anyway," was 29% for the red group compared to 14% for the purple group. These statistics indicate that the red group was more willing to explore links even if their initial perception was unclear about the usefulness of those links in furthering their understanding of the case issues. Consequently the heterarchical case structure may have caused the participants to meaningfully reflect on the usefulness of the case content by exploring what may have been perceived as irrelevant information in order to appropriately frame the issues of the case.

The descriptive data also indicated that the purple group had a response rate of 20% compared to 11% for the red group on reason (a), "link perceived as very helpful for case analysis," and a response rate of 50% compared to 29% in the red group for reason (b), "link perceived as somewhat helpful." These two statistics imply that students interacting with the hierarchical case structure perceived more links as useful or helpful for case analysis than the group interacting with the heterarchical structure. However the nature of the hierarchical structure (i.e., the topical and layered-like organization of the links) necessitates that students "dig deeper" into the structure to find information, which could explain why exploration of the case was more comprehensive by the purple group. Interestingly though, the links that both groups found very helpful for case analysis (reason (a)) were very similar. Both groups found that malpractice cases, articles on informed consent and malpractice lawsuits, patient/physic ian interviews, and legal forms were very helpful in understanding the case issues (Table 1). This finding shows that both groups identified similar case information as useful but suggests possibly that the purple group may have been aided by the topical structure of the case design to identify relevant case information whereas the red group had to more carefully reason through the usefulness of the embedded links to identify relevant case information.

Research Question 2: How does collaboration between group members in the heterarchical hypermedia case compare to collaboration in the hierarchical hypermedia case?

In terms of collaboration between group members, the data from the computer tracking program showed that 97% of the decisions made by the red group were group-based compared to 86% in the purple group. This finding was also confirmed by the interview data. In response to interview question 5, the participants in the red group indicated that their strategy was "discussion mostly, sharing our perspectives, giving arguments for why this is important, and making decisions on which links to choose based on which one would help us figure out what informed consent was." The purple group did not seem to be as engaged collaboratively as the red group. In response to question 5, the participants in the purple group indicated that they "just wanted a summary ... my mission was to get a summary, specific grasp on positions." There was no evidence in the interview data of the purple group of discussion on how to evaluate the usefulness of the case content in formulating problem issues (Table 3).

Research Question 3: How did students in each of the groups perceive the learning task in each of these case designs? Was the learning task perceived as authentic, challenging, complex, ill structured, meaningful, relevant, and engaging?

The interview data confirmed that both groups perceived the learning task (the case) as ill-defined, authentic, engaging, and meaningful. In response to interview question 1, the red group indicated that the case was "very ill-structured" and "very authentic ... when you think of an instructional design project, you walk in there and don't know the problem, don't know the resources, etc." and "in authentic cases even if you know some of those things, you don't have the opportunity to get those things in the order that you want" and "also some of the links provided information that wasn't helpful and that's real life." The purple group indicated that the case was "definitely ill-defined ... it's up to the student to explore and find out what the problem is on their own" and "we didn't know what the problem was even after going through some of the links" and "it wasn't until after going through some links and gathering information that we tried to make a hypothesis as to what the problem was."

In response to interview question 2 (what did you think of the topic of informed consent), the red group commented that the topic of informed consent was very timely and engaging "because people are currently having problems with HMOs (Health Maintenance Organizations) and they are not well informed when they go to surgery." Similarly, the purple group commented that the topic was "engaging because at one point of our lives we are going to have to talk to doctors about possible surgeries and sign forms ... so the topic was timely and relevant." See Table 3 for more interview results supporting the findings for this research question.

Research Question 4: How did students perceive the case structure in each of the groups? Did they find it difficult to navigate through the case? Why or why not? Did they find it difficult to locate specific information?

In terms of case structure, the red group reported that the case was "structured very well" and suggested the addition of more pictures, audio, and video. The red group also reported that "there were major arteries (themes) in the case" that prompted a trail of exploration based on the story-like nature of the case content: "For example, on the administrative meeting page, we were introduced as instructional designers and then oh my god we have all these options, we could go to the office, the library, talk to Allen. That was a huge one, it revealed a lot of information." In response to interview question 4 (was it easy to find what you were looking for), the red group reported that it was not easy to locate information because "we didn't know what we were looking for and just like real life, this led us to have a lot of collaborative discussions ... it was a matter of filtering out, and that filtering out prompted collaborative discussion." In response to interview question 8 (how would you change the case) the red group suggested leaving it as is except for the addition of more multimedia as previously mentioned.

The purple group (hierarchical case structure) reported similar comments overall with some worthy-to-note variations. In response to interview question 4, the purple group reported that it was difficult to find information because they did not know what they were looking for. This was consistent with what the red group reported. In response to interview question 8, the participants in the purple group suggested (a) that hyperlink labels should be more contextualized and paraphrased to aid with the navigation; (b) that layers should be reduced; and (c) that information should be consolidated in the form of summaries at the first level of the hierarchical structure. In response to interview question 3, one participant in the purple group complained about the navigation: "my biggest beef was the navigation ... I felt that the information should have been consolidated ... it was very frustrating however the general structure was good." However the other two participants in the purple group described the hierarchi cal case structure as "fine" and added that they liked how "things were spread around" and "how the problem scenario was the first thing to read and then there was a second layer with the other case perspectives, and so forth."

These findings suggest that the heterarchical case structure may have been more appealing overall than the hierarchical case structure due to its embedded and more contextualized (story-like) linking structure and nonlayered organization. More specifically, the finding that one student in the hierarchical case design complained about having to go through layers to get to information confirms the structural differences in the two case designs and suggests that top-down linking structures may prevent students from making meaningful and well-reasoned choices about how to explore case content. It is important to note here that the student from the purple group who was frustrated with the layered design of the hierarchical case was the operator for the group.

Another interesting finding was that two of the students in the purple group commented that they would rather go through this case on their own and come together as a group afterwards to discuss issues. One participant felt that "this is an exploration thing" and that "a user should be able to go where they want to go at any time without being restricted or guided by others." The other participant added "this is a computer thing...I don't share computers well with others...that's why we all have our own computers." These comments further support why collaboration was not as strong in this group as it was in the red group.

The following is a summary of the results previously discussed:

* Exploration in the hierarchical case design was found to be more comprehensive in terms of the number of links visited/revisited, number of unique links visited, and the total number of interactions generated, however, the average time spent on a page in the heterarchical case design was one and a half times the average time spent on a page in the hierarchical case design.

* Both groups identified the same links/pages as most useful with respect to relevancy to problem-solving suggesting that both hypermedia designs cued the learners to relevant case content, however, the group interacting with the heterarchical case perceived more links initially as useful and later determined that these links were not useful, which may imply that discriminating between relevant and irrelevant case information in a heterarchical design could be more difficult than in a hierarchical design.

* The heterarchical case design triggered more collaboration between group members than the hierarchical case design. The red group's strategy was clearly one of discussion and reasoning through the problem information.

* Perceptions of the ill-structuredness, authenticity, meaningfulness, and real-world relevance of the learning task were equal in both groups.

* Perceptions of case structure in terms of organization of links, resource information, and navigation, were overall more positive for the heterarchical case design with the exception of the comment about adding more multimedia elements to the case.

The two additional data sources (problem solutions and researcher's notes) not only confirmed the findings, but also revealed some inconsistencies that need to be addressed in future research designs. In terms of the problem solutions, which were in outline form, the red group indicated that the key issue was largely a communication problem between doctors and patients regarding informed consent and that there are no clear methods to evaluate that patients fully understand the risks and side effects of preoperative and operative gall bladder surgery. Their recommended solution was that doctors need to inform patients "completely by providing resources and talking about all issues" and that patients should be given specific methods to obtain information and gain knowledge about "how to ask doctors questions and how to research their medical problem." The purple group was not as clear in their formulation of the case issues and problem solution. Although the group indicated that the case issue was largely one o f communication between doctors and patients regarding informed consent, they did not suggest strategies on how to address this issue. Their focus was more on revising the actual forms of informed consent so that patients can understand better what they are consenting to.

Both groups did not clearly identify the ID issue, which was assessment of patients' knowledge about the risks and side effects of preoperative and operative surgery. The case solution calls for the development of appropriate evaluation and assessment methods that guarantee patients' knowledge about the risks of surgery to minimize malpractice litigations against doctors and hospitals. Additionally, the case solution should include the design of training procedures and instructional modules that ensure that doctors and/or hospital personnel are effectively communicating the necessary information to patients so that patients can make informed decisions. The development of instructional modules or information sessions for patients should also be recommended. It is important to note here that this research study did not emphasize or focus on the problem solutions. Its primary focus was on evaluating students' perceptions about hypermedia case designs of ill-structured problems and on tracking students' journeys and interactions with the case to study the nature of the independent variable, case structure. Important information was revealed that would help chart the course for the next research study.

The information revealed, particularly the additional time spent by the red group on the case, and the stronger collaboration demonstrated, justifies why their solution was perhaps more cogent than the other group's solution. There was however one inconsistency noted in the researcher's notes (and revealed in the tracking data as well) and that is that the purple group did not click on the "discussion link" which contained the actual problem question in the hierarchical case design. This may have impacted the problem solution of the purple group. This finding suggests that information in hierarchical design structures is more compartmentalized and prioritized according to layers and menu structures and could potentially lead to students' missing out on important case information unless it is clearly delineated. To add more confluence to this issue, the students in the purple group (hierarchical group) did not perceive the organization to be more isolated or layered. They noted that other than the problem scen ario, which was the first link they visited, the other links in the topics sub-menus seemed to be of equal importance. This is an inconsistency that needs to be further addressed by enhancing the design of the hierarchical case to ensure that students perceive it as a layered top-down structure and that they do not miss out on important case information.

FUTURE IMPLICATIONS

This exploratory case study revealed important information about web-based hypermedia case structure and its impact on student learning in a problem-based learning environment. Most importantly, it revealed that heterarchical case designs potentially increase student collaboration in a web-based hypermedia design of an ill-structured problem and engage students in thinking critically about case content. However, more research is needed to support and solidify these initial findings and to address the inconsistencies noted. The heterarchical versus hierarchical design needs to be further studied to ensure that depth versus breadth is the delineating factor and that irrelevant and relevant case content can be equally integrated in both case designs. Additionally, different student groups (e.g., low vs. high prior knowledge) should be subjected to hypermedia case studies using several cases with varying structure (heterarchical vs. hierarchical) and varying problem-contexts. Case analysis and solutions need to b e evaluated by experts in the field in terms of their viability as a proposed solution. Evaluation criteria in future research will include answering questions such as "Was the problem solved?," "Can the learners articulate the causal relations implied by the solution?," "How cogent was their argument?," "Does it address all of the issues and constituents?," "Did the learners effectively reflect their own domain knowledge?," and so forth. (Jonassen, 1997, p. 86). The experts will be asked to rate their evaluations using such parameters on a Likert scale to quantify the problem solutions. Case analysis and solutions would need to be more comprehensive (i.e., not limited to a one page outline of case issues). Students will also be asked to respond in a written format to a list of questions after submitting their case analysis and solutions and will be individually interviewed (instead of a group interview as was done in this study) to inter-rate participants' responses and determine response frequencies. The qu estions and interviews will address problem design issues, problem-solving issues, and group processes.

Since there will be multiple cases with varying structure and problem contexts in future research designs that identical and different groups of students will be interacting with, there will be two layers of data analysis, case-by-case analysis and cross-case analysis, affording a measure of knowledge transfer (identical groups interacting with different case-contexts). This type of research design is known as a multiple-case study design where the unit of analysis is a single case (Winegardner, 2001). In addition to being a multiple-case study design, the second phase will also involve a more defined experimental research design that will test the effect of case structure of ill-structured problems on complex problem-solving skills and knowledge transfer using several learning variables as predictors. This research will lead to the development of a design model that (a) supports the generation and web-based hypermedia representation of ill-structured problem-solving tasks across multiple disciplines, and (b) ensures learning gains in complex problem-solving skills and knowledge transfer. Future findings will also inform, and possibly improve, assessment methodologies in educational research.

Table 1

Relation of Research Questions to Primary Data Sources

Research Questions           Tracking Program
                              Data Source

(1) How does exploration     Number of links visited,
in the heterarchical hyper-  number or links revisited,
media case compare to        number of interactions
exploration in the hierar-   initiated, time spent
chical hypermedia case?      on each interaction,
Is one more comprehen-       type of interaction,
sive than the other?         total session time, per-
                             centage of links per-
                             ceived useful before
                             and after visiting.
                             Why was that interaction
                             selected (relevancy to
                             problem-solving process)?
                             (a) Very helpful for case
                             analysis? (b) Somewhat
                             helpful? (c) Not sure, but
                             decided to explore anyway?
                             (d) Didn't have any other
                             option? (e) Other reason?

(2) How does collaboration   Whose decision was it to
between group members        initiate the interaction
in the heterarchical         with case content
hypermedia case              (group versus individual)?
compare to collaboration
in the hierarchical hyper-
media case?


(3) How did students in
each of the groups perceive
the learning task in each
of these case designs?
Ill-defined? Well-defined?








(4) How did students
 perceive the case
 structure in each of the
 groups?











Research Questions           Interview Questions
                               Data Source

(1) How does exploration     Question 6: Do you
in the heterarchical hyper-  think you had enough
media case compare to        information to identify
exploration in the hierar-   the main issues/
chical hypermedia case?      problems to solve the
Is one more comprehen-       case?
sive than the other?



                             Question 7: Do you
                             think all of the
                             information presented
                             in this case was useful?






(2) How does collaboration   Question 5: Describe
between group members        the general strategy
in the heterarchical         you adopted when
hypermedia case              using the system or
compare to collaboration     navigating through the
in the hierarchical hyper-   case. In other words,
media case?                  did you have a plan of
                             attack?

(3) How did students in      Question 1: What did you
each of the groups perceive   think of the case in
the learning task in each     general? Was it easy to
of these case designs?        understand? Complex?
Ill-defined? Well-defined?
                             Question 2: What did you
                              think of the topic of
                              informed consent? Was it
                              realistic? Relevant?
                              Meaningful? Was it a timely
                              topic? Did it engage you in
                              the case?

(4) How did students         Question 3: What did you
 perceive the case            think of the structure of
 structure in each of the     the case? The way it was
 groups?                      laid out? The navigation?
                              The organization of the
                              links and resources?

                             Question 4: Was it easy to
                              find what you were looking
                              for?

                             Question 8: How would you
                              change the case structure
                              or present it differently?

Table 2

Descriptive Statistics of Tracking Data

Tracking Program                Red Group--
Data                            Heterarchical Case
                                Design

Time spent on case              2 hours and 40 minutes
Number of links visited         33 out of 81 (41%)
Number of revisited links       11 out of 33 (33%)
Number of unique links visited  22 out of a possible
                                81 (27%)
Total number of interactions    34
with the case (mouse clicks)    -  33 links
                                -  1 text selection

Average time spent on
a page/link                     3.04 minutes

Decisions made by group         33 out of 34 (97%)

Reason for interaction:-        -  (a) 3 times, 11%
(a) Very helpful for case       -  (b) 8 times, 29%
analysis; (b) Somewhat          -  (c) 8 times, 29%
helpful; (c) Not sure, but      -  (d) 4 times, 14%
decided to explore anyway;      -  (e) 5 times, 18%
(d) Didn't have any other
option; (e) Other reason.

Percentage of links per-        -  55%
ceived useful for unde-
rstanding the case and
were actually useful

Percentage of links per-        -  31%
ceived useful for under-
standing the case but
turned out not useful

Case links/pages found          * Cases page - contained
very helpful                    the malpractice lawsuits
                                filed by patients
                                * Informed Consent Page -
                                contained the definitions
                                of informed consent

                                * Research Page -
                                contained links to informa-
                                tion about the history of
                                informed consent, articles
                                pertaining to the issue of
                                informed consent and mal-
                                practice lawsuits, and sample
                                statments filed with a
                                hospital regarding claims



                                * Group was unable to
                                access audio interviews










Tracking Program                Purple Group--
Data                            Hierarchical
                                Case Design

Time spent on case              2 hours and 7 minutes
Number of links visited         79 out of 146 (54%)
Number of revisited links       31 out of 79 (39%)
Number of unique links visited  48 out of a possible
                                146 (32%)
Total number of interactions    84
with the case (mouse clicks)    -  79 links
                                -  5 text selections

Average time spent on
a page/link                     1.96 minutes

Decisions made by group         72 out of 84 (86%)

Reason for interaction:-        -  (a) 17 times, 20%
(a) Very helpful for case       -  (b) 42 times, 50%
analysis; (b) Somewhat          -  (c) 12 times, 14%
helpful; (c) Not sure, but      -  (d) 10 times, 12%
decided to explore anyway;      -  (e) 3 times, 4%
(d) Didn't have any other
option; (e) Other reason.

Percentage of links per-        -  55%
ceived useful for unde-
rstanding the case and
were actually useful

Percentage of links per-        -  27%
ceived useful for under-
standing the case but
turned out not useful

Case links/pages found          * Cases page -
very helpful                    contained the
                                malpractice lawsuits
                                filed by patients

                                * Articles page -
                                contained articles
                                from scholarly
                                journals about
                                malpractice issues of
                                informed consenT

                                * Interviews page -
                                contained interviews
                                with doctors, medical
                                staff and patients
                                about informed
                                consent


                                * Legal Forms-
                                contained sample
                                forms required by the
                                US dept. of health
                                and hospitals for
                                informed consent

                                * Group did not
                                 access discussion
                                 page link

Table 3

Interview Results

Interview Question            Red Group-
                              Heterarchical Case
                              Design

(1) What did you think of     Cecilia--definitely very
the case in general? (easy    ill-structured
to understand, complex,       Beth--Very authentic
ill-defined, well-defined)    when you think of an
                              instructional design
                              project, you walk in
                              there and don't know the
                              problem, don't now
                              resources, etc.
                              Kathy--In ill-structured
                              and authentic cases even
                              if you know some of those
                              things that you want, you
                              don't have the
                              opportunity to get those
                              things in the order you
                              want, you don't have
                              the opportunity to get
                              those things in the order
                              you want, person's not
                              available, e-mails not
                              available, etc. Also,
                              some of the links
                              provided information
                              that wasn't helpful and
                              that's real life.












(2) What did you think        Kathy--Very timely because
of the topic (informed        in times of engaging
consent)?                     society and time in the
                              medical age and HMOs,
                              people having problems
                              and not being informed
                              Cecilia--That applies to a
                              real world example, my
                              father goes to surgery quite
                              a bit and he comes back
                              with options and we grill him
                              about pros and cons, who
                              knows what he was told,
                              did you sign a form
























(3) What did you think        Kathy - Think it was
of the structure of the       structured very well.
case? (well structured,       I liked it cause it made
ill-structured)               it an ill structured
                              problem. I wouldn't put
                              more structure to it,
                              perhaps less text, some
                              audio, more multimedia,
                              got tired of reading the
                              text even though I needed
                              to get through it, might've
                              been other ways to
                              present some of the
                              material. I'm thinking
                              images, audio clips, or
                              movie clips might've
                              made it more interesting,
                              Cecilia -There were major
                              arteries and you have a
                              couple of options off of it
                              so we follow those trails
                              to the end and then had
                              to go back. For example,
                              on the administrative meeting
                              page, we were introduced
                              as Instructional designers
                              and then oh my god we
                              have all these options,
                              we could go to the office,
                              the library, talk to Allen.
                              That was a huge one,
                              it revealed a lot of
                              information.




































(4) Was it easy to find       Kathy - Not really, we
what you were looking for?    didn't know what we
                              were looking for. It
                              wasn't like every link
                              gave us an "ah hah"
                              moment. Just like real
                              life, that's what led us
                              to have a lot of collabor-
                              ative discussions and
                              different perspective
                              like 'this is not useful', etc.
                              Cecilia - It was a matter of
                              filtering out and that filtering
                              out that prompted collabora-
                              tive discussion.












(5) Describe your general     Kathy - Discussion mostly,
strategy while using the      sharing our perspectives,
system.                       giving arguments for
                              why this is important.
                              My strategy was to gather
                              as much information as
                              possible, information that
                              has something to do with
                              informed consent, patient,
                              and gallbladder. I would
                              like to look at as many
                              links as possible. Making
                              decisions on which links
                              to choose based on which
                              one would help us figure
                              out what informed consent
                              was.

(6) Do you think you          Kathy - Yes I think, I was
had enough information        glad that we stopped when
to identify the main issues/  we did, I had to pull myself
problems in the case or to    back and reflect on what
solve the case?               we looked cause you got
                              so immersed in all the mater-
                              ials, had to stop and look
                              at what we had but I felt
                              good we had a definition
                              of what informed consent
                              was.
                              Cecilia - I think so cause
                              towards the end we had
                              got to the point where
                              we were thinking is this
                              really a problem of informed
                              consent or of the doctors,
                              ... it wasn't a problem of the
                              signing of the forms, it
                              was of the doctors communi-
                              cating to patients, and the
                              patients educating them-
                              selves and being guided
                              toward resources.
                              able to frame it better. It
                              took to the end for us to
                              realize that. We had to
                              step back and go step by
                              step what we learned,
                              what we gathered, and
                              realize that wasn't really
                              what the problem was
                              and reframe it. It was
                              more a communication
                              issue.

(7) Do you think the          Ceclia - some of the
information was useful?       information was not useful,
                              example the severity, we
                              never went there...so it's
                              not clear. Like the diagnosis,
                              it confirmed what the doctor
                              said but it didn't really help
                              us. In the end, all information
                              helped me come up with a
                              conclusion, even the unuse-
                              ful information was useful.
                              This is the context of the
                              authentic environment.
                              Only when looking at the
                              information after we had come
                              up with a conclusion, we could
                              say that this part wasn't
                              extremely important, but not
                              before.













(8) How would you change      Beth - more multimedia, pictures,
the case or present it        sounds, some thing to pull them in,
differently to students?      graphics.
                              Cecilia - I would leave it as it
                              is. If its objective is to be
                              problem based, authentic ill
                              structured environment then
                              leave it as it is ... the group
                              was beneficial, collaboration
                              was key, if I had to do this
                              individually, I would've gone
                              on with my informed consent
                              focus and been less likely to
                              look at other links that Beth
                              suggested. Maybe we had
                              arguments about where to go,
                              what does that mean, but this
                              keeps you on task when you're with
                              others and helps you flesh out
                              detail and question things and not
                              jump on the first link that you
                              see. If no one is there to hold
                              you back then you're just going
                              to click, click and click. If
                              there is a group someone says
                              something.










Interview Question            Purple Group--
                              Hierarchical Case
                              Design

(1) What did you think of     Dana-I thought it was ill
the case in general? (easy    defined, cause at first
to understand, complex,       we didn't know that the
ill-defined, well-defined)    problem was, and even
                              after going through some
                              of the links we still
                              didn't have an idea of
                              what the problem was
                              that we were trying to
                              solve. It wasn't until
                              after going through some
                              links and gathering
                              information that we
                              tried to make a
                              hypothesis as to what
                              the problem was.
                              Claire-Agree it was
                              ill-defined, had a grasp
                              of the informed consent
                              issue from the initial
                              problem statement, but
                              I know there was
                              something there that we
                              missed, I have a
                              feeling there
                              were links we
                              should've clicked on
                              rather than what we
                              did.

                              Frankie--definitely ill-
                              defined, it's up to the
                              student to explore
                              and find out what the
                              problem is on their
                              own and which is a
                              good thing.

(2) What did you think        Frankie--Engaging to
of the topic (informed        me cause at one
consent)?                     point in our lives we
                              are going to have to
                              talk to doctors about
                              our own health and
                              possible surgeries we
                              may have, so I think
                              its relevant to almost
                              anybody and easy to
                              understand. I had a
                              narrow perspective of
                              what informed
                              consent was but after
                              going through the
                              case study I can see
                              how it can be applied
                              in different situations.
                              Dana--very relevant
                              for us cause we all
                              will need to go to the
                              doctors and sign the
                              forms. It was good to
                              see disparity of
                              information that is
                              provided to some
                              patients, might be
                              able to see that
                              You're going to a
                              doctor who is not
                              providing you with all
                              the right information,
                              so you need to find
                              another doctor.
                              Claire - Thought it
                              was timely and
                              relevant.

(3) What did you think        Claire - My
of the structure of the       biggest beef
case? (well structured,       was the
ill-structured)               navigation. We
                              went to patient
                              information
                              first to get that
                              perspective,
                              question 1
                              then back,
                              question 2
                              then back, felt
                              that the
                              information
                              should've been
                              consolidated,
                              or a previous
                              and next link.
                              It was very
                              frustrating.
                              However the
                              general
                              structure was
                              good. in
                              wondering if
                              there mightve
                              been something
                              more obvious. I liked
                              how the problem
                              scenario at the top
                              was the first thing to
                              read, having a
                              second layer with a
                              patient, doctor,
                              hospital administra
                              tor, insurance and
                              have all of those
                              views there and have
                              supplemental information
                              under those.
                              So could make more
                              apples to oranges
                              comparison.
                              Frankie - I thought
                              the structure was
                              fine. Things were
                              spread around and it
                              wasn't a linear thing,
                              visit this first, second,
                              third, which may be
                              frustrating to some
                              users. You didn't
                              have to give much
                              thought to where
                              things were.
                              Dana - I agree it
                              wasn't linear but there
                              was a theme around
                              all the topics but
                              didn't know what it
                              was until you went
                              into the links. The
                              only link that I didn't
                              know how it tied was
                              the videos. Didn't
                              know how the video
                              of the gallbladder tied
                              in.

(4) Was it easy to find
what you were looking for?





                              Frankie - Only when
                              we knew what we
                              were looking for,
                              then we knew to go
                              back to it for different
                              things, but it wasn't at
                              the main menu, only
                              when you visited that
                              place before.
                              Dana - I agree, kind
                              of didn't know what
                              type of information
                              you were going to
                              get, for example
                              when you went to
                              patient information
                              you got survey
                              questions given to
                              patients.

(5) Describe your general     Clare - Just wanted to
strategy while using the      I knew there were
system.                       players my mission
                              was to get a summary
                              specific grasp on the
                              positions, we found
                              that information was
                              too specific so we
                              went on a hunt for
                              general information
                              and that led us to
                              legal information. So
                              this was good.





(6) Do you think you          Frankie - I think we
had enough information        had enough informa
to identify the main issues/  tion to identify the
problems in the case or to    main issues but
solve the case?               because it's an ill-
                              defined problem there
                              is no 1 solution, so I
                              don't know if you'll
                              ever solve the
                              problem, you may get
                              an idea on perspec-
                              tives, come up with
                              guesses or ideas.
                              Clare - Got a good
                              grasp on the informed
                              consent problem, got
                              the sense there is a
                              specific gall bladder
                              case that had an
                              informed consent
                              issue bent around it
                              and that was just
                              illustrative of this
                              bigger issue, that we
                              didn't get to if it does
                              exist.










(7) Do you think the          Frankie - I think it was
information was useful?       very useful at getting
                              the different perspec-
                              tives for the problem
                              of informed consent
                              specifically the
                              articles, had scholarly
                              articles, then miss
                              magazine gives you
                              down to earth
                              perspective, in that
                              sense that was good
                              information.
                              Dana - Information was useful,
                              with the legal cases you would
                              click on the case and it would
                              bring up that information.


                              Clare - We questioned the legal
                              cases, woman's son was suing the
                              doctor cause medication caused
                              pancreas disease, never
                              mentioned if doctor warned her,
                              needed more information about
                              informed consent here. Survey
                              questions and answers,
                              particularly patient questions
                              and answers seemed too
                              detailed. A summary would've
                              been better.

(8) How would you change      Dana - Label the links
the case or present it        differently, be more specific,
differently to students?      instead of questioni," paraphrase
                              the question that is asked so
                              person knows where they are
                              going and whether information
                              will be useful, instead of
                              going through layers and
                              layers and finding information
                              wasn't useful.

                              Clare - I agree, descriptions
                              so we know what we're getting
                              into. Consolidating information
                              so there are summaries. Brief
                              summaries of legal cases, went to
                              abstract, history...help bring
                              that information to the first
                              level so you don't have to
                              click back again.
                              Frankie - I would prefer to go
                              through this case on my own and
                              come together afterwards to discuss
                              issues. I think as we were going
                              through we discussed where to go
                              but I don't think we collaborated
                              on the issue in general of informed
                              consent. In an exploration thing, I
                              think a user should be able to go
                              where they want to go at any time.
                              Clare - I agree with Frankie. I
                              don't share computers well with
                              others. Maybe it's a computer
                              thing. That's why we all have our
                              own computers.

References

Barrows, H.S. (1985). How to design a problem-based curriculum for the pre-clinical years. Springer Series on Medical Education. New York: Springer.

Barrows, H.S., & Tamblyn, R.M. (1980). Problem-based learning. New York: Springer.

Booth-Sweeney, L. (2001). When a butterfly sneezes: A guide for helping kids explore interconnections in our world through favorite stories. Williston, VT: Pegasus Communications.

Brown, J.S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42.

Dabbagh, N., Jonassen, D.H., Yueh, H.P., Samouilova, M. (2000). Assessing a Problem-based learning approach in an introductory instructional design course: A case study. Performance Improvement Quarterly, 13(3).

Denzin, N.K. (1978). The research act: A theoretic introduction to sociological methods. New York: McGraw-Hill.

Duffy, T., & Cunningham, D. (1996). Constructivism: Implications for the design and delivery of instruction. In D.H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon and Schuster Macmillan.

Fitzgerald, G.E., & Semrau, L.P. (1996). Enhancing teacher problem-solving skills in behavioral disorders through multimedia case studies. Presented at American Educational Research Association's Annual Meeting, New York, 1996.

Gerdes, D. (1998). IMSA center for problem-based learning. Illinois Mathematics and Science Academy. [Online]. Available: http://www.imsa.edu/team/cpbl/cpbl.html

Grissom, S., & Koschmann, T. (1995). Hypermedia without programming: Automatic generation of presentation documents for case-based instruction. Proceedings of the National Educational Computing Conference, NECC 1995, Baltimore, MD.

Hemstreet, S. (1997). Using hypermedia to facilitate problem-based learning. Retrieved March 25, 2002 from: http://www.edb.utexas.edu/mmresearch/Students97/Hemstreet/index.htm

Herreid, C.F., & Schiller, N.A. (October, 2001). National center for case study teaching in science. SUNY Buffalo [Online]. Available: http://ublib.buffalo.edu/libraries/projects/cases/case.html

Jacobson, M.J., & Spiro, R.J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation. Journal of Educational Computing Research, 12 (4), 301-333.

Jacobson, M.J., Maouri, C., Mishra, P., & Kolar, C. (1996). Learning with Hypertext Learning Environments: Theory, Design, and Research. Journal of Educational Multimedia and Hypermedia. 5 (3/4), 239-281.

Jonassen, D.H. (1997). Instructional design model for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development. 45(1).

Jonassen, D.H. (2000). Toward a design theory of problem-solving. Educational Technology Research and Development, 48(4).

Kagan, D. (1993). Contexts for the use of classroom cases. American Educational Research Journal, 30(4), 703-723.

Keep, C., & McLaughlin, T. (1995). The rhetoric of link traversal. The Electronic labyrinth. Hypertext Fiction Research Group [Online]. Available: http://jefferson.village.virginia.edu/elab/hf10044.html.

Larson, K., & Czerwinski, M. (1998). Web page design: Implications of memory, structure and scent for information retrieval. In Proceedings of CHI '98, Human Factors in Computing Systems (LA, April 21-23, 1998), ACM press, 25-32. Copyright (c) 1998 by ACM, Inc.

Last, D., O'Donnell, A.M., & Kelly, A.E. (2001). The effects of prior knowledge and goal strength on the use of hypertext. Journal of Educational Multimedia and Hypermedia. 10(1), 3-25.

Lesh, R., Hoover, M., Hole, B., Kelly, A., & Post, T. (2000). Principles for developing thought-revealing activities for students and teachers. In A.E. Kelly & R.A. Lesh (Eds.) Handbook of research design in mathematics and science education (pp. 591- 645). Mahwah, NJ: Lawrence Eribaum.

Norman, G.R., & Schmidt, H.G. (1992). The psychological basis of problem-based learning: A review of the evidence. Academic Medicine, 67(9), 557-565.

Oliver, K.M. (1996). A critical analysis of hypermedia and virtual learning environments. (ERIC Document Reproduction Service No. ED 412 925)

Patton, M.Q. (1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage.

Robson, C. (1993). Real world research: A resource for social scientists and practitioner-researchers. London: Blackwell.

Rogers, P.L., & Erickson, M. (1998). Layers of navigation: Hypermedia design for an ill-structured domain. (ERIC Document Reproduction Service No. ED 423 857)

Schon, D.A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass.

Shoffner, B. & Dalton, D. (1997, February). Applying problem-based instructional strategies to returning adult learner populations. Paper presented at the annual meeting of the Association for Educational Communications and Technology, Albuquerque, NM.

Siegel, M., Derry, S., Kim, J., Steinkuehler, C., Street, J., Canty, N., Fassnacht, C., Hewson, K., Hmelo, C., & Spiro, R. (2000). Promoting teachers' flexible use of the learning sciences through case-based problem-solving on the WWW: A theoretical design approach. In B. Fishman & S. O'Connor-Divelbiss (Eds.), Proceedings of the Fourth International Conference of the Learning Sciences (pp. 273-279). Mahwah, NJ: Erlbaum.

Stepien, W., & Gallagher, S. (1993). Problem-based learning: As authentic as it gets. Educational Leadership. 50(7), 25-28.

Sutyak, J.P., Lebeau, R.B., & O'Donnell, A.M. (1998, June). Unstructured cases in case-based learning benefit students with primary care career preferences. The American Journal of Surgery, 175.

Yin, R.K. (1993). Applications of case study research. Newbury Park, CA: Sage.

Yin, R.K. (1994). Case study research: Design and methods, (2nd ed.). Thousand Oaks, CA: Sage.

Winegardner, K.E. The case study method of scholarly research. The Graduate School of America. Retrieved July, 2001 from: http://www.tgsa.edu/online/cybrary/case1.html