Teaching reflection: information seeking and evaluation in a digital library environment.

Abstract

In this article, we explore influences of electronic information systems on teaching methods. Data are analyzed from the information literacy portion of an introductory computer science course. This curriculum was taught by the first author, whose pedagogical goal was to teach students to search for and evaluate information using a variety of systems. Her teaching methods were informed by three complementary theories: Kuhlthau's (1993a) process model, cognitive flexibility theory, and situated cognition. She also employed Schon's (1983) reflective practitioner model, which stipulates that teachers evaluate their pedagogical methods as a course is in session. Although this work is far from being completed, we have confirmed that teachers must be able to reflect on specific incidents and adjust their teaching methods according to individual situations rather than strictly follow prescribed models. Even though the new information systems encourage interaction and offer user-friendly interfaces, the ability to search effectively across systems and critically evaluate retrieved information still needs to be taught. In other words, the digital library environment demands instructional intervention which is flexible and responsive to the situation. Thus we perceive digital libraries as systems comprised of the user, digitized information and software tools, and human mediators.

Introduction

Finding information in libraries has never been particularly easy for young people, even in a library world dominated by print. Can we expect the process to be any easier in an electronic library world? Familiar models have been supplanted by new ones which represent entirely new paradigms. Information systems now have interactive potential -- they are dynamic entities disengaged from earlier limiting parameters. How do we teach students to recognize the cues necessary for survival in digital library environments, What do they need to understand in order to negotiate these spaces with purpose and skill?

Our discussion is based on an examination of the first author's experiences teaching the information literacy portion of a required secondary level introductory computer science course. The purpose was to teach students to be effective seekers and users of information in an electronic environment and to understand the relationship of these skills to searching for information in other contexts. Jacobson also introduced online communication netiquitte and broader Internet-related ethical topics such as privacy, and censorship.

Jacobson's goal was to teach search processes (the traditional focus of much bibliographic instruction and library and information science research) and to focus on evaluation of information beyond basic determination of relevance. As other researchers have emphasized (e.g., Jacobson & Martin, 1993), the librarian's job is to teach students to critically evaluate and know what to do with information as well as to find it. hroughout their coursework, Jacobson's students were asked to pass judgment on the quality of various information items using a rather specific, yet flexible, set of teacher-suggested criteria that would be tailored to their evolving needs.

Jacobson employed teaching methods that she hoped would "scaffold" (i.e., support and structure student learning), prompt, and enable students to self-regulate in an unpredictable environment. Simultaneously, she followed the reflective practitioner tradition described by Schon (1983) to scrutinize her own teaching. Schon stated that practitioners have a tacit knowledge of their field and often reflect on their own experiences "to cope with the unique, uncertain, and conflicted situations of practice" (Schon, 1983, p. ix.). This approach has been consciously adapted as a technique to improve practice, particularly in the field of education (see, for example, Nelson & Smith, 1995). Jacobson engaged her students in classroom activities that would be revealing of their thought processes, and therefore would be more informative of the efficacy of her teaching.

Although two semesters were spent gathering and analyzing data. this project is far from being complete. This article is therefore exploratory and does not represent a comprehensive description of what occurs in a digital library environment. However, our preliminary research has shown us that traditional teaching methods must be flexible, not only because of individual differences among students, but because the electronic environment changes the structure, practices, and culture of information seeking. Further observations and interviews need to be conducted to more fully understand how digital library environments affect the cognitive processes of both students and teachers as well as the classroom culture.

Theories which Informed the Teaching Methods

Jacobson's teaching and these subsequent analyses were heavily influenced by three areas of research: Kuhlthau's process model, cognitive flexibility theory, and situated cognition or situated learning. All three theoretical frameworks share a constructivist outlook on learning; they agree that it is possible to view an event from many perspectives, that meaning and experience are inextricably, linked. and that learners "construct" rather than "receive" knowledge.

Kuhlthau (1988) proposed a model for information searching which contains six stages: task initiation, topic selection, prefocus exploration, focus formulation, information collection, and search closure. Each of these stages is characterized by particular sets of feelings, thoughts, and actions. In subsequent research, she and her colleagues elaborated on this theme, addressing the cognitive and affective symptoms students commonly experience in each stage (Kuhlthau et al., 1990). A central thesis of this body of research is that information searching is a process -- a continuum of overlapping events, not a discrete set of isolated incremental steps. Early stages of the process are typified by feelings of confusion, uncertainty, and anxiety, which students assume afflicts them individually and not their fellow students. Kuhlthau (1988) proposed that these frustrations are alleviated when students are taught to expect these feelings and "to tolerate inconsistent and incompatible ideas" (p. 240). This resiliency releases them from a paralysis that prevents progress.

Kuhlthau (1993a) described the dichotomy between the highly ordered universe of librarianship with its systems for collecting and organizing information and the highly irregular universe of student information needs. For example, a high school history student who is assigned to select a modern political situation that compares to one from the Middle Ages will not find a ready made subject heading for such a topic in a library catalog. Kuhlthau argued for the recognition of a "principle of uncertainty" for library services which acknowledges this dichotomy as well as the feelings of frustration and anxiety that so many library users experience (Kuhlthau, 1993b). She suggested zones of intervention that correspond with stages in the information search process and offered a series of specific intervention strategies to use in an information search process approach to instruction (Kuhlthau, 1994, 1996, this issue of Library Trends). These strategies include:

1. Collaborating -- the librarian or peer acts as a collaborator, which also situates the search process in a nonisolating context more typical of real world information seeking tasks.

2. Continuing -- Intervention is a continuous process because information problems are not static.

3. Conversing -- Conversation not only elicits more informed help from the librarian/counselor and feedback from peers but also helps students articulate and understand their information problem and, ultimately, to develop a metacognitive sense of where they are in a process.

4. Charting -- Charting is a system of using visual representations such as conceptual maps to manage and organize large or seemingly vague ideas, to recognize patterns and relationships, and to stimulate a cohesive sense of direction.

5. Composing -- Kuhlthau (1994, p. 71; 1996, p. 102) uses the example of journal writing which, she says, promotes reflection, formulation, and the development of constructs.

Another concept of learning is cognitive flexibility theory (Spiro et al., 1988) which examines advanced stage learning -- i.e., the period between initial exposure and practiced expertise. The theorists argue that teachers tend to oversimplify and decontextualize complex instructional content, precisely at the stage in which students need exposure to variability and nuance. Consequently, students are unable to transfer knowledge and apply it to new situations. Cognitive flexibility theory offers a number of prescriptions to combat this phenomenon, including using multiple knowledge representations (such as analogies, case examples, or lines of argument); explicitly linking abstract concepts to case examples; stressing the interrelated and web-like nature of knowledge (rather than compartmentalizing it); introducing complexity early in small, cognitively manageable units; and promoting knowledge assembly from various previously learned knowledge components (rather than stressing the intact recall of previously memorized information).

In testing this theoretical perspective, researchers have developed elaborate electronic hypertext environments which afford multiple exposures to a variety of rich cases with links to underlying themes. Learners also have opportunities to add their own perspectives to the cases and links. The design of these environments is intended to "criss-cross" conceptual landscapes, enabling the perception of multiple paths and interrelationships (see, for example, Nelson & Smith, 1994; Jacobson & Spiro, 1995).

Kuhlthau's process model and cognitive flexibility theory share some common views concerning the nature of learning in ill-structured domains such as information retrieval. They agree that the learning process is not linear -- that it is complex and requires multiple exposures, pauses for reflection, and opportunities for reiteration with alternative strategies and views. Both perspectives advocate teaching students to anticipate and therefore to be prepared for inconsistency. A principal tenet of cognitive flexibility theory is that "ill-structured domains are best thought of as evincing multiple truths: Single perspectives are not false, they are inadequate" (Spiro et al., 1992, p. 122). There is an insistence that knowledge which will be used in many ways must be represented and taught in many ways. Finally, Kuhlthau's perspective adds the dimension of the influence of affective factors -- the impact of the learners' feelings upon the ability to learn new knowledge. Her work also addresses the unique aspects of learning a process, which is a more apt description of the nature of searching for information, rather than learning a domain or distinct body of content knowledge.

Situated cognition (Brown et al., 1989) is based on the premise that knowledge is situated within a social setting and is profoundly influenced by that context and its culture. Rather than "acquiring" knowledge as though it were an objective artifact, learners accrue knowledge through the enculturation of social cognitive structures. In a teaching situation, situated cognition often employs an apprenticeship model and generally involves collaborative activity. An instructor models a task by demonstrating the strategies that are used, then coaches or provides cognitive scaffolding in the form of structured support during multiple practice sessions, and finally fades back as students become more independent. Instructional design should support knowledge-rich "authentic" situations and provide opportunities for students to be involved in social interactions and conversations that help support the process of enculturation.

Situated cognition assumes a number of outcomes. Collective problem solving promotes synergistic insights that may not occur in individual problem solving. In groups, students display multiple roles, allowing them to articulate their knowledge, reflect constructively on their performance, and confront ineffective strategies and misconceptions. Finally, collaborative work is a hallmark of most real-world situations.

Situated cognition adds a social-cultural dimension to cognitive flexibility theory and Kuhlthau's process model. Each perspective also makes use of narrative as a form of transmitting or internalizing knowledge. Stories are key elements in the social construction of knowledge (McLellan, 1994). Kuhlthau's use of journals and other composing techniques also draws on the use of narrative. She noted that: "The strategies of talking, writing, and thinking seemed to be as important to students as the actual sources they used" (Kuhlthau, 1988, p. 237). Finally, the use of cases and analogies in cognitive flexibility theory constitute another form of narrative.

In sum, elements from all three paradigms combine synergistically to illuminate aspects of the intellect, the affective domain, and the influence of the outside world. The multiple theoretical viewpoints were used to inform Jacobson's pedagogy as well as our analyses of student work (see also Goodnow & Warton, 1992; Jacobson & Jacobson, 1993). We were also quite curious about the application of these perspectives to a digital library environment.

Description of Course and Data Analysis

During the 1995-1996 year, the University Laboratory High School(*) experimented with a new format for its introductory computer science course. Intending to fashion a more general computer literacy curriculum, the course was taught by a team of four teachers from computer science, journalism, and the library to reflect a broad range of computer applications. The course was taught to approximately 130 students including all of the "subfreshmen" (a combination of seventh and eighth graders), all the freshmen, and a selection of upper class students. The focus was on the use of computers as tools for creativity, communications, and information organization. It included such topics as the use of standard software applications, a basic exposure to machine language programming, and desktop publishing. The year culminated in students preparing and presenting a major project on a theme of their choice.

(*) University Laboratory High School is a secondary level public laboratory high school for high ability students at the University of Illinois at Urbana-champaign.

Students were expected to both learn from and teach each other. Many of the assignments involved group work and/or electronic communication. Most of the major projects were done in groups, and students were required to describe how tasks were divided, who developed expertise in which areas, and what outside resources they used. Students were encouraged to work on homework assignments together reflecting the fact that real world workers collaborate and that it is not considered cheating. As a new course and one which was taught by four different teachers, there were rough spots that certainly contributed to feelings of anxiety for some students but, in the long run, did not seem to have a significant impact on most student learning.

Jacobson's role in the course was to teach students to be effective seekers and users of information in an electronic environment and to cover netiquette and Internet ethics issues. Information skills were broadly defined to include the responsible gathering and sharing of information. Jacobson also made an effort to integrate content from students' other courses into her assignments and lectures. She designed most of her assignments for the subfreshmen to support their research for a long-term science class research project. These students were to follow a cutting edge topic of their choosing throughout the school year. In the Fall, they wrote a background paper on the topic and subsequently were required to keep up with the current literature and submit periodic bibliographic updates. They finished in the Spring with a final report summarizing changes in the field over the year. In her extensive work with students in their science class, Jacobson provided more traditional bibliographic instruction in the library setting with the science teacher always present. From time to time, however, the students were confused about which role she was playing -- librarian, computer science teacher, or science teacher -- and in which class they would get credit for activity done under her tutelage. She saw this confusion as an advantage because she was able to leverage the computer skills and the library skills as information and communication skills situated within a meaningful context.

For this project, we examined Jacobson's descriptions of the assignments in chronological sequence along with her written reflections in order to understand how the course developed over time. We also studied all of the available completed assignments and Jacobson's comments on student papers. The only assignments that were not analyzed were those which were not archived, such as the verbal exercises that were done in class, or those that had to be returned to students before they could be photocopied. The students' identities were kept confidential. Although students did not keep journals, we had access to other forms of narrative records (e.g., e-mail correspondence, essays written for class, etc.). We analyzed the material using qualitative techniques, which allowed us to carefully scrutinize changes in teaching and to learn about concerns of a digital library environment.

Appendix A is provided to give an overview of the sequence of Jacobson's assignments, the pedagogical intent of each one, and the general way in which each reflects the influence of the theoretical perspectives described earlier. Rather than explicate the assignments individually, we chose instead to focus a more detailed critical lens on a selection of five assignments which follow in the next section. The chart is intended to be referenced as the section is read, particularly in terms of linking application with the theoretical perspectives.

Teaching Reflection

Finding Subject Terms in Book Catalogs

Description. The purpose of this assignment was to have students practice identifying and testing subject terms in library catalogs. It was also intended to give them a reason to use e-mail, to start them on a path of articulating and reflecting on search strategy, and to foster collaboration and peer feedback in the information search process. The assignment coincided with procedural instruction in the use of e-mail which was taught by other members of the computer literacy teaching team. Jacobson's class presentation consisted of netiquette instruction and a quick overview of the assignment using a printed copy distributed in class (see Appendix B). The class session version of the assignment was an application of modeling. The example answers were intended to convey that a wide variety of responses was acceptable. For example, "I got nothing" was a legitimate response because, in library research, nothing is often what one finds. The example also modeled a strategy of culling a bibliographic record for clues that can strengthen a search. Students were required to make two search attempts to instill the idea of process and to emphasize the use of multiple tactics. The actual assignment was e-mailed to students who were divided into small working groups (in the case of the subfreshmen, organized by shared cutting edge topic areas). Students were to send their responses to the members of their group with a copy to Jacobson. After a specified period of time, they were to reply with feedback to each person in the group.

Observations. Because this assignment was the first serious use of e-mail for most students, many of them were not yet very adept with the protocol. A number of them had difficulty with the nuances of replying, sending a copy of a message, and sending a note to multiple recipients. They needed more experience with this well-structured domain to be able to take on the ill-structured one of searching for information. The feedback piece of the assignment presented an additional obstacle. Many students simply forgot about sending feedback and, when they were reminded, could not complete it because they had deleted the original messages from their peers. At this age (or any age!), students have a hard time keeping track of events that are to occur several steps into the future. This experience also reminded Jacobson that students were not accustomed to collaborating, especially through e-mail. However, she learned that e-mail use and collaborating with peers were different activities that needed to be explicitly taught and scaffolded into the students' culture as well.

This instructional activity did help many students combat, as Kuhlthau notes (1994), the sense of being alone in their confusion. The following verbatim interchange is an interesting example (the numbers correspond to the questions asked in the assignment):

G.L.:

1. My topic is immunity [sic] to antibiotics. I chose it because I have experienced it first hand and thought it was kind a cool, but aggravating [sic].

2. I used, you guessed it, the Uni Library

3. I picked option one and searched for antibiotics and imunity [sic].

4. Nothing!

5. I chose option two with the same words.

6. Nothing!

7. I also searched for several other things including medicine -- way to [sic] many things but nothing, imunity [sic] -- nothing, atibiotics [sic] -- a book on fungus and lichens.

9. Obviously this was not very usefull [sic) in the aspect I got nothing, except now I know this is going to be a lot harder than I thought....

A response from P.T. (a student colleague): you spelled immunity wrong, mabye [sicl that's why you didn't find anything under that. And, I agree, this is going to be a lot harder than I thought.

Empathy was a strong element in the subfreshmen dialogues, which were full of good luck exchanges and pleas for help. P.T did identify a likely reason for G.L.'s failure to find information, but she also took care to respond to his feelings about the failure. The older students tended to be much more analytical, if not always correct, in their analyses. What follows is an example of the frustration of a freshman who was unsuccessful in his search for a book on the "Phoenecians and Sea Peoples" for a history paper:

This search was useless. I think that it failed for two principal

reasons...

The Uni High book catalogue is a kind of stupid system. Many of

the books on ancient Greece probably have references to the

Phoenicians that would be useful, but the word Phoenician wouldn't

ring w/the system because the book is not primarily about

Phoenicians and so it wouldn't be listed as a major heading on the

information card in the heading. It seems that to answer this

problem all I would have to do would be to search under ancient Greece.

However, if I do this I'm very likely to get books that don't have

anything [on] Phoenicians as well as ones that do and I would have

to go [on] a wild duck chase.

Thus, the two reasons for the failure of this search are the

narrowness of my topic and the inefficiency of the Uni High book

catalogue. I think the answer to my problem is to go to a bigger library.

Not just because they'd have a better computer system, but also

because they'd have more books and I could use a narrow topic to get

a good selection of books.

When this student received no responses from his peers, Jacobson felt that it was incumbent upon her to take a turn in the conversation. His assessment that information could be found in more general books about ancient Greece was a correct one, as was his observation that Phoenicians would therefore not be listed "as a major heading on the information card." However, this absence was not a weakness in the computer system but a characteristic of indexing practice which would hold true in any size library. A search for information in a larger library probably would be more fruitful, due instead to the size of the collection rather than to the inadequacies of the computer system. She reassured him that his topic was not too narrow or otherwise inadequate.

Cutting Edge E-mail Assignment

Description. The science teacher was very interested in giving the students a sense of the communication cycle in science. In her curriculum, most student work was shared with the group rather than being the traditional closed conversation between teacher and student. The cutting edge theme itself was an exercise in understanding the nature of scientific communication and progress -- i.e., how an idea takes hold, becomes developed and shared, changes in response to new influences, and so on. She also wanted students to understand that scientists often work in groups, that they pass on references to one another, and share work-in-progress. She had always supplied the students with a list of each other's topics as a way of encouraging them to share relevant bibliographic findings and mimic this expert behavior. Though they worked individually, most students had selected topics within a rather narrow and predictable range of shared interests.

Together the science teacher and Jacobson formalized a way to model and encourage this type of cooperation. During a science class session, Jacobson assigned students to send two e-mail messages over the course of a school quarter (approximately eight weeks) to "colleagues" whenever they found something to share. These instructions were distributed in class:

Send a minimum of two e-mail messages to a colleague or colleagues

with suggestions or experiences to share, such as:

* Terms that worked for you (send these only to others in your group)

* Good articles you found that might be of use to others

* Other resources you've found that might be of use to others

* Difficulties you are having that others might be able to help you

with (be specific!)

It will probably be most fruitful to write to people in your cutting

edge group, but do not hesitate to send helpful information to

students in other groups. A list of students and their topics is on the

back side of this sheet. Colleagues must be in the subfreshman

class ;-)

IMPORTANT: A copy of each message you send must also be sent to

Ms. Morris (pmorris) and Ms. Jacobson (jacobson). Heres an

example of how to send a copy to more than one person when you are

using Pine:

To :csmith, jdoe, dwhite

Cc :pmorris, jacobson

Attchmnt:

Subject : Great cutting edge article for our group

--Message Text--

Hi! I just found a great article you guys might be interested in too.

I found it by looking under "libraries, digital" after having no luck

looking under "digital libraries" in Readers Guide. Anyway, the

article is called "Evaluating Digital Libraries in the Context of

Learning and Teaching" by F.F. Jacobson and E. Ignacio. Its on page 27

in the October 1995 issue of Cutting Edge Libraries.

The language of the assignment was intended to assure the multiplicity of possible information sources -- indexes, bibliographies, colleagues, etc. -- and to reconfirm that there was no one single correct way to locate relevant information. And this particular activity acknowledged what is known about how practicing scientists actually conduct their own literature reviews (Garvey & Griffith, 1980; Wolek & Griffith, 1980; Crane, 1972).

Observations. For a number of reasons, our idealistic expectations were not quite fulfilled. With such a long lead time before the due date, many students forgot about the assignment. Their work was turned in at the last minute, hardly reflecting the spontaneity of scientists exchanging e-mail between distant labs. Similar to the first assignment, the students were not collaboration-minded, instead prioritizing their assignments by deadline. Accustomed to operating in a competitive and individualistic environment which ranked them linearly (i.e., grades, etc.), the practice of collaboration seemed out of place.

Also, with students in such close physical proximity to one another, sending e-mail often felt like a cumbersome unnecessary step to them. It probably would have been more appropriate to ask them to tape notes to each others' lockers (their de facto labs), saving them the trouble of finding a free computer and the time to check their e-mail. However, there were a few exhilarating moments when a student would rush excitedly into the library to follow up on a colleague's tip. The instructors at least felt that they were giving students a message they didn't hear too often -- that there are advantages to cooperating rather than competing in isolation.

The content of students' advice to each other consisted primarily of references to URLs and bibliographic citations for periodical articles. Visuals in Web sites were particularly valued. Students expressed some confusion about whether or not Internet resources would count in science class (they did) and, since Web searching had not been formally introduced yet in the computer literacy class, there were some interesting descriptions of the process. Here is one student's portrayal of hypertext:

I was on Netscape and I found a very good page on Environmental

issues. It is packed with info, but is kind of thick, having no pictures

(at least that I could see). It also has an internal citation thing where

it has a number that you click on to see the bibliography...

I was on Netscape again, and I found an extremely useful page. It

had some facts, then it said that if you wanted more, you should

click "here". It would tell you the size of the text and pictures you

would get if you clicked "here". It had a lot of good pictures and

information.

Some students supplied dates and times of lectures on campus. A few alerted others against specific Web sites they felt were inappropriate in some way -- too much advertising, too technical, too undeveloped, etc. Others asked for follow up information (e.g., "Remember the article that I gave you? ... I hope it helped!"). Some people asked for general help on a topic from those who were doing similar work. This was a strategy that was suggested by Jacobson in cases where students were having particular trouble finding resources -- and also as a model exhibited in the scientific community.

The tone and content of the messages revealed interesting patterns and raised some questions. E-mail was treated as the equivalent of phone messages or informal letters -- i.e., general conversation peppered with side annotations and personal identifiers similar to the patter on newsgroups ("Let the funk be wich ya! -- P-Funk --"). Even under the guise of exchanging scientific information (that their teachers would also receive carbon copies of), students did not give much heed to grammar or spelling. One student stated that she found an article that may have "teeny tiny potential. Then again maybe not." Certainly the real situation for them was one in which they were talking with friends and not distant professional colleagues. Both the boys and the girls closed their messages with smiley faces or with slang signoffs like "cya" or "c-ya later," using a tone more typical of the notes kids pass each other in high school. On the other hand, e-mail seems to lend itself to informal discourse, even in scholarly circles (Eisenberg, 1994). This leads to a different but interesting research question: How is communication between professional colleagues affected by e-mail?

If a message was particularly laden with information, the tendency to evoke informality was even more noticeable. Perhaps students did not want their friends to think that they were taking the assignment too seriously and thus risk social disapproval. They may have found it personally inappropriate to talk to their friends in a stuffy manner in any medium. Or they may have had enough experience with e-mail by this time to regard it as a space for fun and were trying to sustain that ambience.

Along with maintaining the casual tone, the correspondents rarely provided much context to their messages. Only one student left the original message in a reply. Another student contributed context by stating that he found a lot of information on __, "which is your topic." Many just wrote "thanx!" and left it at that. One of these writers had a signature file which contained the entire lyrics of a Leonard Cohen song, many times longer than her one-word message. The students seemed to treat e-mail as a delayed conversation rather than as formal letters being passed back and forth. The ease of deleting original messages probably added to this sense of informality. The lack of context may also have been due to their dismissing the exercise as a school assignment. One student asked: "Do I get credit even if [my e-mails] say the same thing?" Truly spontaneous messages would not include copies to instructors, though knowing that their teachers would see the messages seemed to have no impact on care in spelling and grammar.

Some students seemed to feel that e-mail was not necessarily a reliable form of communication and preferred to deliver their information in person. Instead of posting a URL or a citation in an e-mail message, these students used the message to request a face-to-face meeting. One student, however, copied the articles he found on the Web directly into his e-mail message, so the recipient did not have to go anywhere. In contrast, three students in the same group all photocopied articles for each other and sent e-mail messages like this one: "I found an article in Scientific American Magazine on genetic engeneering [sic]. I don't know if it will be helpful or not. I photocopied it for you, so remind me to give it to you." One student wanted to make sure that the information he sent to another student was received and requested e-mail (rather than face-to-face) confirmation. Another student wrote "This is #2" either to make sure that the original message got there or to ensure that the recipient knew there were two messages. In sum, much of the dialogue reflected a hybrid style of communication which incorporated elements of paper-based and telephone media along with the less trusted electronic medium.

Evaluating Web Sources

Description. The purpose of this exercise was to promote a habit of information evaluation, specifically by using cues from the search process itself. It coincided with students being introduced to HTML programming taught by the journalism member of the team. Jacobson first spent a class session moving through multiple examples -- comparing and contrasting search techniques in a traditional library catalog with search techniques on the World Wide Web. She used the Library of Congress Subject Headings as a way of introducing the concept of controlled vocabulary and a regulated system of information retrieval. Students pondered the use of such terms as "Xenotransplantation" for species-to-species organ transplants and "cookery" for cookbooks. With souvenir cards from the old card catalog, they dissected the elements of a catalog record and used these clues to infer information about the potential usefulness of a book.

Next Jacobson took students through a guided search of the Web. They discussed the entrepreneurial and independent nature of search engines (comparing the omnipresent advertising to a fantasy of ads popping up for every tenth bibliographic record in a library catalog) and pondered how these characteristics might influence the outcome of a search. The elements of search engine summary records were also dissected, again in terms of what they could inform searchers about the potential usefulness of the sites. The students looked for cues that would help determine length, depth, and purpose in a similar fashion to the cues provided by bibliographic records. How could the students tell which short record would link to someone's vita or personal home page; What elements could supply hints that would prevent unnecessary links to those ubiquitous under construction apologies? They laughed at the analogy to finding a library book with nothing in it but an introductory paragraph. The fact that the students were creating their own home pages perhaps provided the best case example of all. The process for Web publishing was abundantly transparent to them and dismantled most automatic assumptions of any Web site's authority or magical attributes.

The assignment, which was completed by e-mail, did not focus so much on searching as it did on site evaluation. Students were to identify a "good" site and a "bad" site on their topic and were also required to articulate the reasons for their selections (see Appendix C). Jacobson realized that by requiring the students to find a "good" site and a "bad" site, she was predefining those as the two available categories.

Observations. As was typical of the other e-mailed assignments, students' messages were characterized by a casual conversational tone. There were many spelling and punctuation errors and a tendency to use colloquial phrases: "This information was not useful, because it just piddled and didn't state facts."

Students characterized "good" sites as having both valuable content as well as links to other useful Web sites. They repeatedly mentioned this winning combination of robust content and good links -- "so much information and also tells you where to get more." A few students noted higher level distinctions. For example, one boy reported that his good Web site provided useful links that hadn't been retrieved on his Lycos search. In other words, he was appreciating the culling and selection efforts of the Web site creators and recognizing their expertise as it compared to the rote processes of a search engine. Yet another student noted that there were several links in his site to FAQs (Frequently Asked Questions files), which he recognized as sources of authoritative and more comprehensive information. We were interested to see that some of the comments about usefulness were more akin to negative positives, damning with faint praise. A site was praised for simply not being an advertisement, causing us to wonder if student expectations of net sources are lower. There were many comments like this: "It was useful because it was specific and not about something totally off the subject I needed. It was about my topic."

The "bad" sites were categorized as being off the topic or peripheral to the topic. The numerous near-off topic sites included many personal home pages, among them institutional home pages that were never meant to be sources of topical content for public use. In some cases, a site was relevant but was rejected for another reason -- i.e., it was too technical, written in a language the student did not speak or, as so often happens, it was incomplete in some way. Just as sites were characterized as being useful if they had links, they were characterized as being not useful if they did not have links. Students were also quite critical of unsubstantiated opinion: "It didn't really have useful information -- just some sarcastic stuff that I really didn't need." There were rare comments like this one: "The un-useful document happened to be the page with the highest [relevancy] score." Very few students articulated this relationship between search engine relevancy rankings and their own assessment of a sites' value. The biggest crime from the students' point of view, however, was the plethora of advertising. Students saw the ads as a kind of net pollution, at least in terms of their own needs.

Though we mentioned that in some ways students seem to have lower expectations of net sources, we also noticed that they were critical of features that are standard in many types of print sources. Bibliographies, directories of U.S. postal mailing addresses (for example, of sports teams), references to secondary information that were not direct links to actual sources (i.e., an e-mail form that would reach the sports team itself), were regarded as useless. It seemed that once students were in the Web environment, they expected it to be consistent and to provide the same kind of connectivity across the board. As one student phrased it: "I was looking for into straight off the netscape."

Encyclopedia Guide

Description. The purpose of this assignment was to have students get involved in a more rigorous evaluative exercise and to begin differentiating between form and content issues. The guide format put them in the teacher role, requiring them to articulate (in essence, by developing their own cases promoting cognitive flexibility) in a way that is needed when communicating to a novice audience. This kind of conversing also allowed them to describe the search process as they discovered it; their composition crystallized their experiences (Kuhlthau, 1996, p. 102).

Jacobson experimented with two versions of this assignment, one for the subfreshmen (see Appendix D) and one for the older students (see Appendix E). The latter version asked students to consider a specific set of criteria in their explanations which provided cognitive scaffolding for their thinking. Concerned that this laundry list might overwhelm the younger students, Jacobson designed a more open-ended assignment for the subfreshmen. The tradeoff of the looser format was that it did not scaffold student thinking and so was less of a teaching tool in and of itself -- at least in terms of teaching the specific criteria Jacobson wanted the students to learn. Also, this version did not specify explicit outcomes, which provide the structured context younger students tend to need. However, the subfreshmen's work did provide an opportunity for us to see how they viewed their experience and which elements of the process they would find important enough to describe.

Observations. The subfreshmen tended to describe every single step needed to find information. It appears that this is how they interpreted the instruction to "teach them how to use these encyclopedias."

Pretend your topic is the ozone layer and ozone

-first double click on the Encarta icon.

-as soon as it loads up click once on the Contents button

-type ozone layer and press return.

-in the article you may find little cameras. If you click on these

they will show you a picture.

-or if you see a little video recorder in the article you can click on it

to watch a movie.

-if you want to see an article on ozone itself go to the top of the

screen and click the menu button once.

-In the menu double click on tide screen.

-select contents again but this time type ozone and press return.

-In the article you may find little cameras. If you click on these

they will show you a picture.

-or if you see a little video recorder in the article you can click on it

to watch a movie.

Time saving, generally described in terms of ease of use, was an important criterion. Only one person felt that print encyclopedias were easier to use; the rest stated that CD-ROM titles were easier and therefore faster to use. In the electronic encyclopedia, you can zoom to related topics much faster without having to find something in a completely different volume of an encyclopedia. Students did draw a distinction when they judged a user interface to be particularly confusing.

In general, most subfreshmen felt that the CD-ROMs had better information. There was a commonly expressed sentiment that detailed fine-tuned information was to be found in print media but more general information was to be found in the electronic titles ("the highlighted text is good stuff"). Jacobson was struck by students' lack of attention to qualitative differences between the titles. For example, very few observed that articles in the Academic American and Encarta were usually briefer than the articles in World Book. Visual information was also very important to the subfreshmen. Without visuals, an article was often dismissed as being boring and not very useful. Only if highly detailed information was needed was the lack of illustration acceptable.

For both print and digital encyclopedias, authority seemed to be measured in terms of the amount of detail found in the articles. If the article covered only the basics, it was not considered to be as reliable as articles that had more in-depth information (with the exception of the unconventional format and high vocabulary level of Britannica, which confused a number of students). Many students included commentary about accessibility and comfort issues. They stated that the advantage of print encyclopedias was that they were more accessible and easier to read. By accessibility, they were generally referring to computer hardware -- the necessity for it, the lines to get to it, and its periodic unreliability. Several complained about the difficulty of reading text on the screen.

A number of students were quite confused by conceptual differences in search techniques. In general, they did not seem to differentiate between searching by topic method or by key word method, unlike the older students who often stated that one must choose between these two options. The subfreshmen either did not know there was a difference or did not think it was an important thing to mention.

The structure of the assignment for the older students supplied them with a much more defined set of criteria to use. They were prompted to describe authority in terms of authorship. They also had to seek out two different strategies to use in looking for information. The subfreshmen were not supplied with this model, and many may not even have been entirely conscious of the possibility of multiple methods.

The freshmen also related strategies step by step but were careful to write about the difference between searching by word or topic. This was probably due to the structure of the assignment; the freshmen were asked to explicitly describe two methods of looking for information (see Appendix E). They also stated that short articles were not as informative but differentiated this characteristic from authority. Pictures were nice and were expected of multimedia encyclopedias, but many students stated that pictures were not enough to satisfy an information need: "Microsoft Encarta has the best pictures out of all the electronic versions. The most useful encyclopedia for our topic was Encyclopedia Britannica [version not specified]."

The freshmen were generally better at articulating encyclopedia characteristics and instructions for their use. They could describe hypertext words as being related to subjects and not just "good stuff." They did not complain in the same way about having to read a great deal of text on the screen, suggesting instead that users could just print it out and take it with them.

The narrative form of the assignment gave students an opportunity to share their own sense of discovery, to be conversational, and to select elements they found to be especially pertinent. Two freshman girls working on the assignment together had this to say about the two versions of World Book (their sample topic was onions):

We managed to find a nice long article in the World Book as well as (surprise! surprise!) a picture. Usually the CD ROM does not have many pictures. But what it does have, you can print out (as we have done; see last page). The article was really easy to read (but not so good for taking notes. i.e., look up at the screen, down at the notes, up at the screen ... etc.)

... Now this (I garuntee [sic]) will be interesting. At least we thought so. While continuing our Encyclopedia Guide project and looking up onions in the regular World Book (under "o") we found that the picture we printed out was exactly the same (!) as the picture in the book. The book managed to outdo the CD ROM with a graph which the latter did not have. The print was smaller and we had to xerox (costing us 10 cents) the picture for you. A definate [sic] minus for the book form. Yet, when the computer labs are full, the book form encyclopedias will be there, always glad to be in use.

Web Sneak Preview 

Description. This in-class assignment pushed the evaluation theme further. Rather than searching, students were to compose Siskel/Ebert style reviews of two Web sites. Of the two sites, one was to be from a list of five serious (i.e., purposely informational) sites, one was to be from a list of five fun (i.e., recreational) sites. In pairs, they filled in the descriptive information -- intended audience, authority (authorship), and a brief description of the site. Individually, they filled out review sheets, expressing opinions on depth of coverage, ease of exploration, usefulness (for its purpose) and accuracy of information, quality of design, and uniqueness (listing characteristics that made the site more than just electronic page turning). Finally, they rated each site on a one-to-five-star scale and gave it a thumbs up or thumbs down designation. On the next class day, the final vote tallies were shared.

Observations. All answers were acceptable as long as they were substantiated. Students were being evaluated on their ability to compose well-considered responses, not on the particular flavor of their final opinions. Jacobson intended this exercise to communicate that diverse opinions are simultaneously valid -- that information is only as good as the degree to which it meets a particular need at a particular time. In other words, in concurrence with cognitive flexibility theory, there is no single answer; there are multiple perspectives on every situation. We were especially interested in seeing how the students would do with describing usefulness of a site in terms of the site's expressed purpose. For example, the most popular "fun" site, Mr. Edible Starchy Tuber Head (http:// winnie.acsu.buffalo.edu/potatoe/), received generally high marks. The site was interactive (visitors could design their own Mr. Potato Head), it had unique information such as the history of the toy, and it provided links to other sources. But some students only found it irritating and could not concede that it met a certain stated function. Students sometimes did and sometimes did not note the unique Web qualities of these sites (i.e., the interactivity of the game site, for example, or the ability to zero in on a map's location on the "Virtual Tourist" site. They generally descr-ibed these attributes but did not necessarily identify them as being unique to the electronic environment-almost as though they were too obvious to mention. We wondered if the students' familiarity with computer games as a kind of electronic standard might have accounted for their blase attitude.

The Reflective Practitioner

It is always difficult to create authentic situations in the classroom setting and to replicate a truly situated or apprenticeship environment. Although the assignments were designed to reflect real life situations, we found that the students were still very much aware of the teacher-student relationship (Schon, 1983), and thus were often more focused on the requirements of an assignment than they were on the situation Jacobson was trying to evoke. The natural tone of their e-mail exchanges notwithstanding, a grade on the assignment was uppermost on their minds. Try as educators might to situate schooling activities within meaningful contexts, the most fundamentally authentic aspect of school life is the inevitable evaluation process.

We saw this tension surface most clearly in the cutting edge information exchange exercise, for which two different parameters were imposed on the students: collaboration and the use of e-mail. The students were placed in groups with others who had similar topics. But because very few of the topics were identical, the information-sharing exercise may have seemed artificial. As was noted earlier, the e-mail medium was not always the most natural one to use with students in close physical proximity to each other (though there are many examples of enthusiasts who use e-mail to communicate with intimates ranging from office mates to spouses!). Clearly, though, we were trying to impose a mode of communication that was not otherwise developing naturally for all students.

Another assignment in which real-life situations were being imposed in the classroom setting was when students were introduced to online newsgroups as a form of information dissemination and exchange. The topic of computer ethics was introduced as a way to situate instruction in Usenet protocol. Jacobson posted six computer ethics scenarios to the school library's newsgroup (which had lain dormant until then) and assigned the students to respond to at least one of the scenarios in the form of a follow-up message to the group. The following is an example of scenario narrative style:

Lester sends e-mail to the entire student body inviting them to a BYOB party at his house while his parents are out of town. Lester receives a message from a system administrator calling him in for a meeting with school officials. He objects because he feels that his e-mail is his own private business.

Students were again primarily concerned with completing the requirements of the assignment. Therefore, they typically responded to the initial scenario posting and did not follow up on their peers' topical threads (for which they would have been given credit, though perhaps this distinction was not apparent to them). As in the Web sneak previews assignment, they were not evaluated on the particular cast of their opinion, but on their ability to articulate a rationale based on what they had learned in class. However, the students were not using the newsgroup in the way newsgroups are designed to function -- i.e., with give-and-take dialogue. Very few of them were aware of their peers' opinions. However, at the same time, older students in the school and a number of alumni (from far-flung locations) spontaneously contributed their own opinions to the dialogue. Their interest was intrinsic, and they were not having to confront the Netnews software environment for the first time. The online dialogue was followed up with an in-class discussion, which drew enthusiastic real time engagement from virtually every student.

Students' developmental differences became key to understanding the need to customize instructional tactics. For example, if Jacobson wanted the subfreshmen to understand authority in terms of authorship on the encyclopedia guide assignment, they needed to be given more explicit instruction on the topic of authority. She had been concerned about mandating a lengthy laundry list of evaluative criteria. The assignment could instead have been subdivided into smaller segments, following a recommendation that cognitive flexibility theory makes to introduce complexity in cognitively manageable chunks. These segments could then culminate in a synthesized final product.

Reactions to a Digital Library Environment

We hold a view of the digital library as having the potential to allow its users a participative role. For example, digitized information can be cut and pasted into personal word processing documents or e-mail messages and otherwise directly manipulated. The digital library is not necessarily static; documents can be revised, responded to, and annotated (Burbules & Bruce, 1995). As a result, we were very interested in seeing what would happen when given the opportunity to encourage more interaction between user and information. We accomplished this by requiring the use of e-mail, introducing newsgroups, and assigning other activities that promoted conversation around text and user. If this interaction can be conceived of as being part of a digital library system, then the digital library is one which can be personalized as well.

In the computer literacy course setting, students' e-mail signature files were a very visible hallmark of personalization. Interesting and varied, we wondered if some students forgot that the files were always present, even in mail sent to their teachers. One student included quotes from her mother ("i'm a tough broad"), Kermit the Frog ("gosh gonzo, we're so glad you're not shrinking anymore"), a teacher ("you guys, stop being jerks"), and a friend ("yeah, I know that'd be like screwing your teddy bear!"). We wondered the same thing about the students who changed their logon names to monikers like "beef guru" and "I Am A Spitting Cobra."

If the Internet is to be regarded as a digital library, which is the perspective we have taken, we learned that students generally regard it as being better than the traditional library. Comments like these on the Web-searching e-mail assignment were typical:

This would be a worthwhile document for my purposes because it has information I might not have been able to find without research in many books which would be a long and tedious search.

It tells you current topics and cutting edge information that you can't find in libraries.

However, students were not at all without criticism of the Internet:

I found a horrible document through the Alta Vista search. It was supposed to be a transweb informational site that gave all the facts and the current legislation on organ procurement, but every time I clicked on one of the headlines, none of the information was there. It is stupid to have a site on the net, but have absolutely nothing on it. It was very frustrating to look on a web site with nothing there. It only makes me mad!

In sum, we noted an interesting contradiction -- a belief in the superiority of the Internet while, at the same, lowered expectations of its resources as compared to traditional library resources. Having created their own Web pages, the students were acutely aware of how easy it was for anyone to become an author. Even if they could not articulate the details of the reviewing processes that go into more conventional publishing formats, they understood the lack of a quality filtering mechanism.

A few students noted that they could acquire unique collections or archives: "It is not really an info page, it is an archive which holds a lot of different games.... worthwhile because it contains hundreds of games that anybody can download." "Lots of neat photos each individually submitted by people .... Pictures are unique... simulates an art museum complete with a lobby and walls of art." An interesting thread we observed was that of students' seeming willingness (particularly the older ones) to express appreciation for a site even though they might not find it useful for their purposes. One student reported that she found an astrology site that would have been useful for a paper because it was mainly information about the mathematics of astrology." Of her bad site she said: "Good site, but not for a paper ... Obviously since it was a page for the daily horoscope, it wasn't a good type of information. I did like that site, however." This was the kind of fine tuned discrimination Jacobson was looking for as an instructor, but it included an unexpected emotive expression. There was also evidence of an engagement that does not seem to occur as often with students in the print world -- a delight in the whimsy of many Internet resources. Many students indicated that they found something they wanted, plus something fun or unexpectedly fascinating, such as the student who was looking for information on clarinets. He found a page full of useful information, helpful links, and -- bonus -- a long string of musician jokes which he promptly printed and shared.

While we are claiming to view the Web as a digital library, students also had exposure to a "real" digital library providing access to a large collection of full-text online resources. Students at Uni (not just those taking the computer literacy course) had an opportunity to use the Electric Library (see Weinberger in this issue of Library Trends) during a three-month trial period. Most students completed surveys from which we were able to glean some interesting insights. In terms of searching tactics, students almost always entered phrases that looked more like search terms or subject headings (ozone depletion, chaos theory, bird behavior, Persian Gulf veterans). It is not clear if this was from habit because they did not believe that sentences would work or because using phrases just struck them as a more efficient way to search. A few did use sentence-like phrases (senseless gun violence, benefits of single-sex education, scurvy on subcellular level).

There was a strong interest in being able to limit searches more precisely than the system appeared to allow, particularly to be able to specify word order. Students were frustrated with the many irrelevant cites, especially when the relevancy ranking of these was high and placed the cites toward the top of the list. It was clear that the students were able to articulate their search needs and were frustrated when they could not because the system did not allow them an obvious way to do so.

The students were over-whelmingly positive in their expressions of appreciation of the service. The chief benefit seemed to be convenience, that: 'It can retrieve just what you want and you can have it without physically running around looking for it." The tone of the comments was palpable relief, as though a great personal weight had been lifted. A great deal of the frustration and anxiety of library work is about the uncertainty of being able to actually put one's hands on the sources. Fighting through the search process is just the first step, a factor library professionals do not always have uppermost in their thoughts. Students did not print as much as we thought they would and, in fact, seemed to be rather selective about it. It is possible that the comfort of knowing that the information was there relieved them from feeling pressed to print everything that was remotely relevant. On the other hand, their reticence to print indiscriminately could just have been due to the fact that the library's printers are somewhat slow.

Conclusion

If students need structured opportunities to reflect on what they are learning, how will that be accomplished in a digital library environment? One lesson that has been confirmed for us is that librarians and teachers cannot follow a prescribed theoretical model in order to teach students to be effective searchers. Instructors must make adjustments as the term progresses and continually be attuned to the learner's development. Teaching requires improvisation and flexibility (Schon, 1983). Strategies also need to be differentiated and tailored to each student's particular needs. The students in this situation learned how to search and evaluate information in a classroom setting. Yet the help they received was also personalized through feedback on assignments, in-class coaching, and out-of-class conferencing. User-friendly digital library interfaces are not enough; skilled mediation and intervention will always be necessary.

In their study of librarians as exemplar intelligent agents, Nardi and O'Day (1996) ask us to consider the concept of an information ecology:

Our vision of a diverse information ecology is that of a richly patterned collaborative system of users, human agents and software agents .... The notion of a "librarian in a box," a reference we have heard in talks on digital libraries, is utterly wrong-headed to us. Rather than seeing human agents and software agents as in competition, as vying for the same place in our world, the wiser course is to leverage the strengths of each, deliberately designing work practices and institutional arrangements that reflect and exploit the possibility of collaboration between human and software agents.

Our own observations have led us to view the digital library as a dynamic system comprised not only of digitized information and software tools but also of users' contributions to the system and their interactions with human professional mediators and peers. In very practical terms, all of these aspects are integral components of the system itself. We have come to the conclusion that our teaching must reflect this holistic view if it is to be successful.