Evaluating Digital Libraries: A Longitudinal and Multifaceted View.ABSTRACT
THE PERSEUS DIGITAL LIBRARY (PDL See page description language.
1. PDL - Page Description Language.
2. PDL - Program Design Language.
3. PDL - Push Down List.
4. PDL - Dave Lebling, one of the co-authors of Zork. ) IS ONE OF THE primary digital resources for the humanities. Under continuous development since 1987, the project has included an ongoing evaluation component that aims to understand the effects of access to digitized source materials Noun 1. source materials - publications from which information is obtained
source - a document (or organization) from which information is obtained; "the reporter had two sources for the story" in the humanities. A summary of the PDL genesis and current status is given and the multifaceted mul·ti·fac·et·ed
Having many facets or aspects. See Synonyms at versatile.
Adj. 1. multifaceted - having many aspects; "a many-sided subject"; "a multifaceted undertaking"; "multifarious interests"; "the multifarious and longitudinal evaluation effort is described. A brief synthesis of results is provided and reflections on the evaluation along with recommendations for DL evaluation are given.
Digital libraries marry the missions, techniques, and cultures of physical libraries with the capabilities and cultures of computing and telecommunications. Evaluating digital libraries is a bit like judging how successful is a marriage. Much depends on how successful the partners are as individuals as well as the emergent emergent /emer·gent/ (e-mer´jent)
1. coming out from a cavity or other part.
2. pertaining to an emergency.
1. coming out from a cavity or other part.
2. coming on suddenly. conditions made possible by the union. All three entities--the two individuals and the gestalt Gestalt (gəshtält`) [Ger.,=form], school of psychology that interprets phenomena as organized wholes rather than as aggregates of distinct parts, maintaining that the whole is greater than the sum of its parts. union--are of course influenced by their context as well. The difficulties arise from the complexity of mutually self-adapting systems interacting in a rich environment. Metrics for success for component parts of a complex system may be distinct from the metrics for success of the marriage (e.g., success for an individual partner is typically necessary but not sufficient to ensure success for the marriage).
Digital libraries (DLs) are extensions and augmentations of physical libraries (Marchionini & Fox, 1999). As extensions, we might evaluate the individual partners using existing techniques and metrics. Assessing the impacts of libraries on the lives of patrons and the larger social milieu mi·lieu
n. pl. mi·lieus or mi·lieux
1. The totality of one's surroundings; an environment.
2. The social setting of a mental patient.
[Fr.] surroundings, environment. are the ultimate goals of evaluation, but the practical difficulties of assessing such complex and varied impacts cause us to measure the effectiveness and efficiencies of library operations and services as surrogates for these impacts. Metrics such as circulation, collection size and growth rate, patron visits, reference questions answered, patron satisfaction, and financial stability may be used to assess physical library performance in this regard. Clearly, these are metrics that may be points of departure for evaluating digital libraries, but they are not sufficient to characterize the new rapidly emerging entity. Evaluation criteria for digital technologies can also be useful. For example, metrics such as response time, storage capacity, transfer rate, user satisfaction, and cost per operation may be useful in assessing technological components but may not be sufficient to characterize DL performance, let alone impact. As extensions of physical libraries and digital technologies, these metrics are good starting points Noun 1. starting point - earliest limiting point
terminus a quo
commencement, get-go, offset, outset, showtime, starting time, beginning, start, kickoff, first - the time at which something is supposed to begin; "they got an early start"; "she knew from the , but we must look further to consider the effects of DLs as augmentations that provide new services, products, and capabilities.
In assessing new services and products, it is difficult to distinguish novelty effects The novelty effect, in the context of Human Performance, is the tendency for performance to initially improve when new technology is instituted, not because of any actual improvement in learning or achievement, but in response to increased interest in the new technology. (both positive and negative) from long-term effects. More importantly, new services and products typically create new effects that cannot be predicted until an "installed base" of practice takes root. Additionally, some of these unanticipated effects are due to the new services and products, and some are due to the marriage of existing services and products to the new ones. It seems certain that assessing these effects will not happen in "Internet time In the early days of the public Internet, Internet time referred to the breakneck speed with which companies scrambled to gain traffic and market share on the Web. A new business could come and go within a matter of weeks. ." The effects of DLs will emerge over time as physical libraries, DLs, and people mutually adapt and mature; the problem of evaluation for DLs is thus one of assessing complex adaptive systems Complex adaptive systems are special cases of complex systems. They are complex in that they are diverse and made up of multiple interconnected elements and adaptive in that they have the capacity to change and learn from experience. .
The goal of this discussion is to provide a view of an important DL that has been evolving for more than a decade. Over this time, both the Perseus Digital Library (PDL) (Crane, n.d.-a) and the related evaluation effort have evolved, guided by central missions to provide and understand the effects of broad access to digitized source materials in the humanities. From the beginning of the evaluation effort in 1987, the primary evaluation aim was to address the impact of this project on users and the humanities community. This article will provide a reflective summary of results for this particular DL, discuss the methodological approaches taken to understand the evolving DL, and argue for multifaceted and longitudinal assessments of DLs in general. The article first describes the genesis, evolution, and current status of the Perseus Digital Library; provides a perspective on evaluation as a research and problem solving problem solving
Process involved in finding a solution to a problem. Many animals routinely solve problems of locomotion, food finding, and shelter through trial and error. endeavor; summarizes how this perspective was applied to the evaluation of the Perseus DL over a twelve-year-period and what outcomes have emerged; and finally provides some reflections and recommendations for DL evaluation.
THE PERSEUS DIGITAL LIBRARY: ACCESS TO PRIMARY RESOURCES
As stated on its Web page: "The Perseus Project The Perseus Project is a digital library project of Tufts University that assembles digital collections of humanities resources. It is hosted by the Department of Classics. is an evolving digital library of resources for the study of the ancient world and beyond." The mission statement reads: "Our primary goal is to bring a wide range of source materials to as large an audience as possible" (Crane, n.d.-b). These themes of evolution and wide-scale access to source materials have been constants from the earliest days of the project. A small team of classicists led by Gregory Crane began planning in 1985, and several small grants supported a number of prototypes that led to a large grant from the Annenberg/CPB Project to begin building the "hypertext hypertext, technique for organizing computer databases or documents to facilitate the nonsequential retrieval of information. Related pieces of information are connected by preestablished or user-created links that allow a user to follow associative trails across the " in 1989. One component of the plan was an external evaluation effort that has continued until the present. The initial plan was to digitize To convert an image or signal into digital code by scanning, tracing on a graphics tablet or using an analog to digital conversion device. 3D objects can be digitized by a device with a mechanical arm that is moved onto all the corners. as many ancient Greek Noun 1. Ancient Greek - the Greek language prior to the Roman Empire
Greek, Hellenic, Hellenic language - the Hellenic branch of the Indo-European family of languages texts and English translations as possible; gather or create images, maps, and video objects related to locations and artifacts artifacts
see specimen artifacts. ; and build tools for searching and manipulating these materials (see Crane, 1988, and Crane & Mylonas, 1988, for early articulations of the Perseus vision). The Apple HyperCard platform was selected since it offered the best hyperlinking and multimedia capabilities in the late 1980s. One objective was to create a CD-ROM CD-ROM: see compact disc.
in full compact disc read-only memory
Type of computer storage medium that is read optically (e.g., by a laser). package that contained many of the primary readings and resources that students taking classics courses would need and to make this package available for the cost of one or two textbooks. In addition to university students, the Perseus team expected that classical scholars Noun 1. classical scholar - a student of ancient Greek and Latin
humanist - a classical scholar or student of the liberal arts
Latinist - a specialist in the Latin language would find the corpus and tools helpful to their research and would also contribute new translations, interpretations, and tools. Because the funding aimed to apply new technologies to improve learning and teaching, the project was characterized as an interactive curriculum.
Elli Mylonas conducted a series of interviews with twenty professors in a variety of humanities fields at twelve institutions in 1987 to discuss how Perseus might be used for instruction (Mylonas, 1987). The results suggested that Perseus could be both a reference and a source of primary materials. Although professors were skeptical about using Perseus for their own research, they saw possibilities for extending and refreshing their own knowledge, especially in small departments where they teach courses outside their main research expertise. They raised issues of using a new technology, navigation in the corpus, and overall quality of the information resources (1) The data and information assets of an organization, department or unit. See data administration.
(2) Another name for the Information Systems (IS) or Information Technology (IT) department. See IT. (e.g., accuracy of texts, keeping information up to date, choice of Loeb editions of texts). Most importantly Adv. 1. most importantly - above and beyond all other consideration; "above all, you must be independent"
above all, most especially , they raised issues about the relationships between Perseus (digital realm) and the rest of the scholarly universe and whether students would limit themselves to Perseus materials and points of view. This issue of defining an information space and shaping user perspective through what is (and is not) included and how users may work through the materials became an important pedagogical ped·a·gog·ic also ped·a·gog·i·cal
1. Of, relating to, or characteristic of pedagogy.
2. Characterized by pedantic formality: a haughty, pedagogic manner. issue for the project. As part of a planning grant before the project began, a workshop, "Assignments in Hypertext," was held at Harvard University Harvard University, mainly at Cambridge, Mass., including Harvard College, the oldest American college. Harvard College
Harvard College, originally for men, was founded in 1636 with a grant from the General Court of the Massachusetts Bay Colony. in March 1988. At that meeting, the Perseus team clarified their aim to focus on primary source material that scholars and students might use to create their own interpretations rather than instructional materials that explicated meaning didactically di·dac·tic also di·dac·ti·cal
1. Intended to instruct.
2. Morally instructive.
3. Inclined to teach or moralize excessively. . This constructivist con·struc·tiv·ism
A movement in modern art originating in Moscow in 1920 and characterized by the use of industrial materials such as glass, sheet metal, and plastic to create nonrepresentational, often geometric objects. philosophy rejected the use of secondary readings and authoritative rigid "paths" through the database(1) and promoted the notion of primary materials as raw materials for student exploration and investigation. To this end, the primary drivers for Perseus became acquiring as much primary source material as possible and developing navigational and analysis tools (e.g., search, hyperlink structures, and morphological mor·phol·o·gy
n. pl. mor·phol·o·gies
a. The branch of biology that deals with the form and structure of organisms without consideration of function.
The first prototypes consisted of HyperCard stacks that presented Greek texts and English translations, graphical site plans for temples and other environments, and basic search and display tools. The staff recognized that rapid changes in technology could render their efforts obsolete unless they chose robust data models. One crucial decision was to use the Standard Generalized Markup Language (language, text) Standard Generalized Markup Language - (SGML) A generic markup language for representing documents. SGML is an International Standard that describes the relationship between a document's content and its structure. (SGML SGML
in full Standard Generalized Markup Language
Markup language for organizing and tagging elements of a document, including headings, paragraphs, tables, and graphics. ) to code texts as they were keyed into digital form. This was an expensive decision at the time because the HyperCard platform did not handle SGML markup (text) markup - In computerised document preparation, a method of adding information to the text indicating the logical components of a document, or instructions for layout of the text on the page or other information which can be interpreted by some automatic system. and the value added Value Added
The enhancement a company gives its product or service before offering the product to customers.
This can either increase the products price or value. by domain experts marking up structure and semantics in the texts was not usable in the short term. In hindsight, this was a prescient pre·scient
1. Of or relating to prescience.
2. Possessing prescience.
[French, from Old French, from Latin praesci decision as systems more than a decade later are able to take advantage of this coding. Another decision was to use an object-oriented approach to managing the multimedia data. A "catalog card" was developed for each physical object (e.g., vase, coin, architectural object, sculpture, site). This card serves as an entry point or reference for all specific files or screens related to that object. For example, an important vase might have more than 100 image files associated (shots at various details around, inside, and on the bottom) with it but one main entry that includes information on the following: Collection, Summary (text), Ware, Shape, Painter, Potter, Context, Region, Date, Period, Dimensions, Primary Citation, Decoration, Graffiti, Inscriptions, Parallels, Collection History, Condition, Shape Description, Sources Used, Keywords, and Views (links to the actual image files). Catalog templates for the other objects are likewise defined (see the art and archaeology collections at http://www.perseus.tufts.edu/ art&arch.html). The Perseus team was thus creating specialized metadata schemes for different objects in the collection long before bibliographic management became known as metadata. These crucial data management decisions were informed by a technical advisory board and an educational advisory board, each made up of internationally prominent researchers who met annually to react to decisions and give advice.
The collection development plan called for amassing a significant portion of the extant ex·tant
1. Still in existence; not destroyed, lost, or extinct: extant manuscripts.
2. Archaic Standing out; projecting. Greek text and a large number of images and maps. Selection of Greek textual materials and translations was driven by logic (useful in undergraduate courses) and opportunity (availability). For texts, in addition to Greek text itself, several other types of information were included: apparatus criticuses (commentaries and explanations added to texts), scholia scho·li·um
n. pl. scho·li·ums or scho·li·a
1. An explanatory note or commentary, as on a Greek or Latin text.
2. A note amplifying a proof or course of reasoning, as in mathematics. (annotations made by people over time on the manuscripts, sometimes becoming part of the text itself), metrical met·ri·cal
1. Of, relating to, or composed in poetic meter: metrical verse; five metrical units in a line.
2. Of or relating to measurement. analysis (for poetry), staging notes (for plays), bibliographies, and English translations. The translations were particularly controversial since there were many translations for the important works, and intellectual property decisions led to using the Loeb translations at Harvard where the project was then based (in a few cases, new translations were commissioned). In addition to these basic files, existing indexes and a lemmatized word index were included. Because the Greek language Greek language, member of the Indo-European family of languages (see Indo-European). It is the language of one of the major civilizations of the world and of one of the greatest literatures of all time. is highly inflected in·flect
v. in·flect·ed, in·flect·ing, in·flects
1. To alter (the voice) in tone or pitch; modulate.
2. Grammar To alter (a word) by inflection.
3. , finding the lemma lemma (lĕm`ə): see theorem.
(logic) lemma - A result already proved, which is needed in the proof of some further result. (root forms) for a word is crucial to reading and translation; as a side effect, the complexity of the morphology morphology
In biology, the study of the size, shape, and structure of organisms in relation to some principle or generalization. Whereas anatomy describes the structure of organisms, morphology explains the shapes and arrangement of parts of organisms in terms of such makes word search in Greek potentially more powerful than in English since there is more information packed into the morphology. Crane developed the Morpheus tool (a morphological parser A routine that analyzes a continuous flow of text-based input and breaks it into its constituent parts. See parse.
(language) parser - An algorithm or program to determine the syntactic structure of a sentence or string of symbols in some language. ) for this purpose, and it was used to create the word indexes and is an important component of the Perseus text analysis tools in the current version. The team identified texts in ten genres (epic poetry Noun 1. epic poetry - poetry celebrating the deeds of some hero
poesy, poetry, verse - literature in metrical form ; Elegiac el·e·gi·ac
1. Of, relating to, or involving elegy or mourning or expressing sorrow for that which is irrecoverably past: an elegiac lament for youthful ideals.
2. , Iambic i·am·bic
Consisting of iambs or characterized by their predominance: iambic pentameter.
1. An iamb.
2. A verse, stanza, or poem written in iambs. and Lyric poetry; tragedy; comedy; historians; orators; mythology; philosophy; inscriptions/papyri; and other). More than ninety primary texts were identified for inclusion in these genres. Greek texts were keyed offshore and then subjected to extensive editorial processing where proofreading Proofreading traditionally means reading a proof copy of a text in order to detect and correct any errors. Modern proofreading often requires reading copy at earlier stages as well. , additional notes, and markup were done. In the current PDL, there are almost 300 Greek and Roman texts in Greek, Latin, or English along with eighteen secondary texts and nineteen Renaissance texts.
An important goal of Perseus is to bring text and other media together to add value to scholarship and learning. The original plan called for both purchasing slides from museums and a large-scale original photography effort. To guide collection development, a set of thirteen topics related to art, architecture, and archaeology (AAA AAA: see American Automobile Association.
(Triple A) A common single-cell battery used in a myriad of electronic devices of all variety. Like its double A (AA) cousin, it provides 1.5 volts of DC power. When used in series, the voltage is multiplied. ) were identified: house, propylaea A Propylaea, Propylea or Propylaia (in Greek — Προπυλαια) is any monumental gateway based on the original Propylaea that serves as the entrance to the Acropolis in Athens. , stadia, stoa, temples and sanctuaries, invention and refinement of architectural idiom, theaters, topography topography (təpŏg`rəfē), description or representation of the features and configuration of land surfaces. Topographic maps use symbols and coloring, with particular attention given to the shape and elevations of terrain. , town planning town planning: see city planning. , artists and artisans, Greek athletics, daily life, and stylistic development in Greek art Greek art, works of art produced in the Aegean basin, a center of artistic activity from very early times (see Aegean civilization). This article covers the art of ancient Greece from its beginnings through the Hellenistic period. . The artifacts that carry the meaning characterized by these topics include: architecture, vase painting, architectural sculpture, other relief sculpture, and freestanding free·stand·ing
Standing or operating independently of anything else: a freestanding bell tower; a freestanding maternity clinic. sculpture (coins were added later). The representations for these artifacts are in the form of slides and drawings that were digitized. Over the years, in addition to rights to existing slides, thousands of original slides were taken at museums and sites in the United States United States, officially United States of America, republic (2005 est. pop. 295,734,000), 3,539,227 sq mi (9,166,598 sq km), North America. The United States is the world's third largest country in population and the fourth largest country in area. and Europe. Original slides were copied and archived remotely, and the acquisition and preservation experience is documented for others to use.(2) Today there are more than 33,000 images available through the image browser, representing more than 500 coins, 1,500 vases, 1,400 sculptures, 180 sites, and 380 buildings--each object having a catalog card entry point.(3)
In addition to the art, architechture, and archeology visual media, there were three meta collections created that cut across the texts and AAA--an encyclopedia, a narrative overview, and an atlas. The encyclopedia is accessible via hyperlink or word search from any view of the PDL. The overview is a substantial essay (an electronic book) by Thomas Martin that introduces the ancient Greek world and includes hyperlinks to items in the DL. The current PDL includes a number of additional secondary treatments (e.g., vase painting, Greek and Latin syntax). The atlas has gone through many changes as it moved from the CD-ROM version that included LandSat imagery and maps for pre-determined regions to the WWW WWW or W3: see World Wide Web.
(World Wide Web) The common host name for a Web server. The "www-dot" prefix on Web addresses is widely used to provide a recognizable way of identifying a Web site. version that is built upon a full geospatial database. The current WWW atlas provides access to more than 1,000 physical places in the ancient world at multiple levels of resolution, ranging from a global view that allows a user to label bodies of water, populated places A populated place is, according to the United States Geological Survey, a city, town, or village characterized by buildings and permanent human population and referenced to its geographic coordinates. , and modern borders, to a zoomed in resolution that allows the user to display contour contour or contour line, line on a topographic map connecting points of equal elevation above or below mean sea level. It is thus a kind of isopleth, or line of equal quantity. lines, spot elevations A point on a map or chart whose elevation is noted. , and rivers (Chavez, 2000).
The Perseus project, from its earliest days, was situated in an academic region (Cambridge, Massachusetts This article is about the city of Cambridge in Massachusetts. For the English university town, see Cambridge, England. For other places, see Cambridge (disambiguation).
Cambridge, Massachusetts is a city in the Greater Boston area of Massachusetts, United States. ) that supplied a wealth of technical and content talent. The project team was led by a philologist phi·lol·o·gy
1. Literary study or classical scholarship.
2. See historical linguistics.
[Middle English philologie, from Latin philologia, love of learning who articulated the mission and assembled an interdisciplinary humanities team that included people with specializations in ancient history, archaeology, art history, and Greek and Latin language Latin language, member of the Italic subfamily of the Indo-European family of languages. Latin was first encountered in ancient times as the language of Latium, the region of central Italy in which Rome is located (see Italic languages). and, over the years, drew graduate student assistants from many departments in the Boston-area universities. Importantly, the Perseus team shared a belief that information technology is a powerful medium for advancing the study and appreciation of the fruits of humanistic hu·man·ist
1. A believer in the principles of humanism.
2. One who is concerned with the interests and welfare of humans.
a. A classical scholar.
b. A student of the liberal arts. thought: and facilitating new levels of expression by students and scholars alike. This point of view was somewhat radical in the mid 1980s, and the original team members who were in untenured faculty positions or about to become assistant professors in classics departments openly discussed the dangers of working to change how classics is practiced and taught. In the short term, these dangers were instantiated, as all three of the tenure-track central team members were not offered tenure in their first faculty position. In the long term, each of these three are tenured ten·ured
Having tenure: tenured civil servants; tenured faculty.
Adj. 1. tenured faculty leading their departments and the field in leveraging technology to advance classics and the humanities in general. There are two lessons here. First, the importance of leadership, tenacity, and commitment and a ready talent pool all contributed to the persistence and evolution of the DL. Second, it is important to assess impact over a long period of time.
The ambitious plan to create Perseus emerged over a three-year period (1985-88) and was supported first by small equipment and planning grants before the four-year grant from the Annenberg CPB/Project began in 1989. The main evaluation plan was developed as part of the proposal for the four-year cycle. Over the 1989-1993 period, the system was developed as a set of HyperCard stacks with a variety of database backend supports on Unix and Macintosh platforms. A CD-ROM version (Perseus 1.0) was produced and published by Yale University Yale University, at New Haven, Conn.; coeducational. Chartered as a collegiate school for men in 1701 largely as a result of the efforts of James Pierpont, it opened at Killingworth (now Clinton) in 1702, moved (1707) to Saybrook (now Old Saybrook), and in 1716 was Press (now out of print). A second Macintosh version (Perseus 2.0) has been available for several years, and a platform-independent CD-ROM version is now available. Funding to extend the system to the platform-independent version, to add materials related to the history of science, to add materials related to ancient Rome Ancient Rome was a civilization that grew from a small agricultural community founded on the Italian Peninsula circa the 9th century BC to a massive empire straddling the Mediterranean Sea. , and to create and evaluate instructional models was obtained from a variety of sources, including the National Science Foundation, the National Endowment for the Humanities National Endowment for the Humanities (NEH)
U.S. independent agency. Founded in 1965, it supports research, education, preservation, and public programs in the humanities. , and the Fund for Improvement of Post-Secondary Education (FIPSE FIPSE Fund for the Improvement of Post-Secondary Education ). These grants extended the hypertext corpus and tools to a more diverse library and made the project an ideal candidate for the Digital Library Initiative Digital Library Initiative - A project to research digital libraries which aims to provide real collections to real users (high school students, University researchers and students, users in public libraries). Phase Two program, which provides support for the 1999-2004 period. Evaluation has been included in all of these efforts at various levels of support to ensure persistent and longitudinal assessment feedback to the project team. Securing a steady stream of funding cannot be overestimated when examining the overall impact of the Perseus DL.
Over the past twelve years, the Years, The
the seven decades of Eleanor Pargiter’s life. [Br. Lit.: Benét, 1109]
See : Time corpus migrated from a HyperCard driven CD-ROM to the World Wide Web while adding new materials and tools. The central mission of providing access to large amounts of source materials has been carried out by the project director (Greg Crane) and many original Perseus team members and they have continued to guide the emerging DL. Although in the early days the project did not refer to itself as a library, the library metaphor was explicitly captured in the catalog card metadata records and in providing cheap and easy access to large volumes of primary source materials. Today the Perseus DL includes more than 225 gigabytes of texts, images, maps, and indexes and garners 300,000 http requests per day mainly at the Tufts site but also at European mirror sites at Oxford and Berlin. Commercial encyclopedias, as well as hundreds of syllabi syl·la·bi
A plural of syllabus. at universities and K-12 institutions around the globe, link to it. A spin-off The situation that arises when a parent corporation organizes a subsidiary corporation, to which it transfers a portion of its assets in exchange for all of the subsidiary's capital stock, which is subsequently transferred to the parent corporation's shareholders. organization, Stoa (www. stoa.org), has been created to support research and electronic publication for humanities scholars, and Tufts university Tufts University, main campus at Medford, Mass.; coeducational; chartered 1852 by Universalists as a college for men. It became a university in 1955. Jackson College, formerly a coordinate undergraduate college for women, merged with the College of Liberal Arts in has begun to support the Perseus DL as part of its overall infrastructure. The DL funding in coming years promises to extend the scope of materials and tools greatly. In the following sections, the evolution of the evaluation effort and key results are detailed.
EVALUATION AS A RESEARCH ACTIVITY
Evaluation has many connotations ranging from highly focused and well-defined product testing to the highest form of cognitive reflection.(4) Classical program evaluation Program evaluation is a formalized approach to studying and assessing projects, policies and program and determining if they 'work'. Program evaluation is used in government and the private sector and it's taught in numerous universities. aims to identify causal models A causal model is an abstract model that uses cause and effect logic to describe the behaviour of a system. See also
tr.v. pre·sup·posed, pre·sup·pos·ing, pre·sup·pos·es
1. To believe or suppose in advance.
2. To require or involve necessarily as an antecedent condition. See Synonyms at presume. causal models. Williams (1986) provides a set of readings that support such qualitative approaches to evaluation. Many theorists propose combining methods through triangulation triangulation: see geodesy.
The use of two known coordinates to determine the location of a third. Used by ship captains for centuries to navigate on the high seas, triangulation is employed in GPS receivers to pinpoint their current location on earth. . Cook and Reichardt (1979) offer a collection of papers that describe qualitative and quantitative approaches to evaluation with an eye toward synthesis, including Campbell's (1979) recommendations for systematic case studies. Rossman and Wilson (1985) provide an example where data from multiple methods are synthesized syn·the·sized
1. Relating to or being an instrument whose sound is modified or augmented by a synthesizer.
2. Relating to or being compositions or a composition performed on synthesizers or synthesized instruments. . In contrast, Bednarz (1985) provides a theoretical overview of the different paradigms and argues that, although it is effective to synthesize To create a whole or complete unit from parts or components. See synthesis. multiple data within a paradigm, synthesizing across paradigms ultimately fails as one approach inevitably dominates. Clearly, evaluation research continues to be an active area of methodological research in its own right.
It is important to distinguish evaluation as a research process from evaluation in the product testing and system efficiency sense. Many specific measures applied to product testing may very well be used as evidence in evaluation research. However, evaluation research considers the interactions of complex phenomena--including people--and reaches conclusions through chains of inferences supported by data rather than direct measurement. As noted earlier, the evaluation literature bristles with debates over basic approaches to evaluation, especially with respect to qualitative versus quantitative methods and rationalistic ra·tion·al·ism
1. Reliance on reason as the best guide for belief and action.
2. Philosophy The theory that the exercise of reason, rather than experience, authority, or spiritual revelation, provides the primary versus hermeneutic her·me·neu·tic also her·me·neu·ti·cal
[Greek herm philosophies. Collecting multiple data sets and triangulating the results is advocated in most paradigms, and the PDL evaluation takes this approach by systematically collecting data using statistical techniques for summarizing data where appropriate but not using inferential statistics inferential statistics
see inferential statistics. to test pre-conceived hypotheses. Rather, triangulation is used to make inferences and develop arguments about PDL meaning and impact. This approach is based on the belief that evaluation is a research process that aims to understand the meaning of some phenomenon situated in a context and the changes that take place as the phenomenon and the context interact. This definition implies that evaluation specify what is the research process (metrics and procedures), what is the phenomenon (its mission and salient characteristics), and the context(s) in which the phenomenon occurs. Of course, when developing evaluation plans and carrying out the work, the primary emphasis is on the research process because the phenomenon and context are so omnipresent om·ni·pres·ent
Present everywhere simultaneously.
[Medieval Latin omnipres to instigators of the evaluation. This last point is the reason outside evaluators are often used, although this implies more expense since the outsiders must "come up to speed" on the phenomenon and context at hand. The Perseus experience suggests that this very process of learning about the phenomenon and the context was itself an important part of the overall evaluation.
Evaluation has both theoretical and practical impact in information science. Theoretical constructs, such as information needs, relevance, and information transfer, are debated and assessed regularly, and metrics for assessing system development and operation are crucial to continued progress in practice. In the case of information retrieval information retrieval
Recovery of information, especially in a database stored in a computer. Two main approaches are matching words in the query against the database index (keyword searching) and traversing the database using hypertext or hypermedia links. , evaluation is often focused on the effectiveness of a result set in a specific search, or aggregations of results across many searches, to assess and compare different search systems (see Harter & Hert, 1997, for a recent review of IR evaluation; see Voorhees & Harmon, 2000, for an overview of the recent Text Retrieval Conference For other uses of "TREC", see TREC.
The Text REtrieval Conference (TREC) is an on-going series of workshops focusing on a list of different information retrieval (IR) research areas, or tracks. [TREC TREC Texas Real Estate Commission
TREC Text Retrieval Conference
TREC Technique de Randonnée Equestre de Compétition
TREC Tropical Research and Education Center
TREC T-cell Receptor Excision Circle
TREC Teachers and Researchers Exploring and Collaborating ] results). Metrics such as recall and precision are typically used, and a standard set of procedures that includes test questions and pooled relevance judgments are used to ensure comparability across systems. Usability testing Usability testing is a means for measuring how well people can use some human-made object (such as a web page, a computer interface, a document, or a device) for its intended purpose, i.e. usability testing measures the usability of the object. is another type of evaluation that focuses on the effects obtained when individuals apply an information processing information processing: see data processing.
Acquisition, recording, organization, retrieval, display, and dissemination of information. Today the term usually refers to computer-based operations. system to accomplish tasks (Nielsen, 1993). Usability testing adopts metrics such as time to completion, accuracy, satisfaction, and errors. A variety of procedures for situating the tasks (laboratory/field setting, assigned/open tasks, and so on) are used in usability tests. Another branch of information science, bibliometrics Bibliometrics is a set of methods used to study or measure texts and information. Citation analysis and content analysis are commonly used bibliometric methods. While bibliometric methods are most often used in the field of library and information science, bibliometrics have wide , aims to assess the impact of individuals or communities (e.g., journals) on research progress through citations and other bibliographic relationships (see White & McCain, 1989, for a review). Citation and co-citation counts (including hypertext links in the WWW) serve as the basic metrics upon which new indicators, such as impact value, are derived. Evaluations are also conducted to determine how effective libraries are in carrying out their missions. Griffiths and King (1991) use a model for evaluating information centers that includes measures in four classes: input cost, outputs (quantities, quality, timeliness, availability, and accessibility), effectiveness (e.g., amount of use, user perceptions of services, user satisfaction), and domain (e.g., patrons, staff, information need types, user behaviors). Saxton (1997) provides a meta-analysis of reference service effectiveness that considers nine variables (expenditures, total collection size, reference collection size, collection size per patron, volumes added per year, volumes discarded, overall change in collection size, proportion of change to total size, and number of hours open). System engineering evaluations judge the effectiveness and efficiencies of hardware and software using such metrics as access and transfer latencies, mean time to failure, and development and maintenance costs. The PDL evaluation depends mainly on educational evaluation Educational evaluation is the evaluation process of characterizing and appraising some aspect/s of an educational process.
There are two common purposes in educational evaluation which are, at times, in conflict with one another. but draws heavily upon each of these information science subfields for metrics and techniques.
THE PERSEUS EVALUATION PLAN AND EVOLUTION
As part of the proposal to the Annenberg/CPB Project in 1988, an evaluation plan was outlined. The plan was detailed during the first six months of the project and served as a guide for activities throughout the four years of funding and beyond to subsequent funding cycles (Marchionini, Neuman, & Morrell, 1989). The plan presented a multifaceted approach to evaluation as a research process that included multiple metrics and methodologies and aimed to understand Perseus as a new electronic phenomenon with impact in multiple contexts. Two contextual factors were strongly influential in slanting slant
v. slant·ed, slant·ing, slants
1. To give a direction other than perpendicular or horizontal to; make diagonal; cause to slope: the evaluation effort toward educational contexts. First, the funding was aimed at educational applications of electronic materials--the project was titled "An Interactive Curriculum on the Ancient Greek World." Second, the evaluation team's background and experience were both rooted in education and instructional technology There are two types of instructional technology: those with a systems approach, and those focusing on sensory technologies.
The definition of instructional technology prepared by the Association for Educational Communications and Technology (AECT) Definitions and Terminology . Steve Ehrmann, the Annenberg/CPB Project program director, asked us to consider the definitional question "What is Perseus?" and, over the years, we posed various explications of the Perseus phenomenon. Crane, in an interview in April 1989, noted that Perseus was "a laboratory" to study heterogeneous information tied together to focus on one subject. To Neuman and me (both faculty in an information science program), it seemed from the start that Perseus was a library that extended the possibilities for self-directed learning.
In the introductory paragraph, the evaluation plan was characterized as a roadmap that would guide decision-making over the years rather than a detailed blueprint specifying all details of the evaluation. This was so for three reasons: technology changes rapidly; variables and metrics related to educational and scholarly processes are complex and difficult to quantify; and we aimed to assess the interactive nature of learning, teaching, and scholarly production. This last point is an important one because assessing interactivity had few precedents at the time and remains a significant challenge today as we struggle to assess browsing and other interactive behaviors in the WWW environment.
The architecture of the evaluation plan was characterized by crossing goals with objects of evaluation to define a set of research questions and then mapping a variety of data collection and analysis methods onto these questions. Three high level goals this DL offered to learners and scholars were access (to large volumes of multiple media source material), freedom (self-directed access and use), and collaboration (among learners and teachers). Four classes of evaluation objects were defined: learners, teachers, the technical system, and the content. These generated a hierarchical set of ninety-four research questions. The learner questions had four main categories(5): specific tactics and strategies with six specific subquestions; overall patterns of use with five specific subquestions; changes of behavior and perception with ten subquestions, one of which also had two subsubquestions; and Perseus use compared to other approaches with three subquestions. Three instructor questions(6) had nine associated subquestions; three system questions(7) had thirty-eight subquestions and subsubquestions; and three content questions(8) had twelve subquestions associated with them. Methods deemed appropriate and applicable were then associated with each of these questions. The basic methodology aimed to collect data using a variety of methods and then triangulating results to answer the research questions. Four classes of data collection methods were defined: observations, interviews, document analysis, and learning analysis.
Five kinds of observations were identified. Baseline observations were made to situate sit·u·ate
tr.v. sit·u·at·ed, sit·u·at·ing, sit·u·ates
1. To place in a certain spot or position; locate.
2. To place under particular circumstances or in a given condition.
adj. the evaluators and build relationships with individuals involved in the observations. These were semi-structured where we sat unobtrusively un·ob·tru·sive
Not undesirably noticeable or blatant; inconspicuous.
unob·tru in classrooms or labs and made notes during lectures, discussions, and lab sessions. Structured observations were defined to follow a specific protocol in a classroom or lab--e.g., systematically observe the behavior and record notes for a purposive pur·po·sive
1. Having or serving a purpose.
2. Purposeful: purposive behavior.
pur sample of individuals. For example, select five students and alternate observations every three minutes "Three Minutes" is the 46th episode of Lost. It is the twenty-second episode of the second season. The episode was directed by Stephen Williams, and written by Edward Kitsis and Adam Horowitz. It first aired on May 17, 2006 on ABC. to record whether they were taking notes, looking at the instructor, and so on. Although we conducted a few such observations, this technique was used less often than we expected due to the difficulty in collecting such fine grained data in a classroom environment or laboratory with so many other pertinent activities underway. Participant observations participant observation,
n a method of qualitative research in which the researcher understands the contex-tual meanings of an event or events through participating and observing as a subject in the research. involved the evaluator with students and were audiotaped. The observer is guided by a semi-structured protocol and may ask or answer questions (participate) according to according to
1. As stated or indicated by; on the authority of: according to historians.
2. In keeping with: according to instructions.
3. the situation. This technique was used heavily in one site where a graduate assistant worked intensively with a class over an entire semester se·mes·ter
One of two divisions of 15 to 18 weeks each of an academic year.
[German, from Latin (cursus) s (Evans, 1993). Think-aloud observations aim to determine what cognitive activity underlies behavior and are used widely in psychology and education research (Ericsson & Simon, 1984). Subjects were asked to think aloud while they worked on various tasks, and the entire session was audiotaped. Both participant observation and think aloud could have been included in the interview category, but we classified them as observations because they are less dependent on self-report on the part of the subject and more focused on the observed activity in which the subject is engaged. The final observation method is automatic screen journaling (transaction log analysis). This technique automatically captures user actions such as keystrokes or mouse clicks, adds time stamps See timestamp. to the record, and may include "snapshots" of the screen at critical junctures of the interaction. Routines for capturing user actions were developed and used extensively in one site over a semester (Evans, 1993). Evans provided graphical displays for sessions that demonstrated both systematic and opportunistic study strategies. Patterns such as clear demarcations between persistent use of texts and images contrast with regular and/or random alternations between media. We have also used transaction analysis to study usage in WWW sites such as the Bureau of Labor Statistics Bureau of Labor Statistics (BLS)
A research agency of the U.S. Department of Labor; it compiles statistics on hours of work, average hourly earnings, employment and unemployment, consumer prices and many other variables. (Hert & Marchionini, 1997). Transaction log analyses are used with the Perseus WWW logs to determine gross interaction patterns today (e.g., number of requests for different resources, temporal patterns of access, and so on).
Interviews were initially defined around course schedules with introductory, midsemester, and exit interviews planned. Although we did conduct interviews at different intervals for some classes, a better way to categorize cat·e·go·rize
tr.v. cat·e·go·rized, cat·e·go·riz·ing, cat·e·go·riz·es
To put into a category or categories; classify.
cat the interviews we eventually used is verbal interviews with individuals or groups and written or online questionnaires With the increasing use of the Internet, online questionnaires have become a popular way of collecting information. The design of an online questionnaire often has an affect how the quality of data gathered. . We often conducted interviews with individual instructors and students. These interviews were guided by semi-structured protocols and were typically audio taped. Such interviews were conducted at ten different universities, in some cases over several years, and have proven to be one of the most valuable evidence sources for our findings and reports over the years. Group interviews were also conducted at several universities. These are likewise guided by general questions and audio taped (one session was videotaped). Most of these sessions were with groups of students and yielded candid commentary on how instructors use Perseus and what students thought about the DL. A written questionnaire was developed for use in classrooms and, over the years, almost 1,000 students at several universities completed the questionnaire (the questionnaire was modified several times over the years to reflect changes in the technology and content). The questionnaire has three main sets of questions: demographics The attributes of people in a particular geographic area. Used for marketing purposes, population, ethnic origins, religion, spoken language, income and age range are examples of demographic data. , including computing and Perseus usage; system features; and impact on learning. Check lists, Likert scales Likert scale A subjective scoring system that allows a person being surveyed to quantify likes and preferences on a 5-point scale, with 1 being the least important, relevant, interesting, most ho-hum, or other, and 5 being most excellent, yeehah important, etc , and a few open-ended questions A closed-ended question is a form of question, which normally can be answered with a simple "yes/no" dichotomous question, a specific simple piece of information, or a selection from multiple choices (multiple-choice question), if one excludes such non-answer responses as dodging a are included. A somewhat surprising result that recurred over the years was the lack of correlation between demographics and learning impact and a positive correlation Noun 1. positive correlation - a correlation in which large values of one variable are associated with large values of the other and small with small; the correlation coefficient is between 0 and +1
direct correlation between system interface features and learning impact. Perceptions about ease of use are closely related to perceived learning effects.
In recent years, as attention has shifted to Perseus impact on teaching and research, a number of verbal protocols and e-mail questionnaires have been used with instructors. In 1999, a short online questionnaire eliciting general information about Perseus use via the WWW was used to collect data. After a pilot test with voluntary responses, the questionnaire was automatically given to every tenth unique visitor to the Perseus home page (IP addresses were recorded so that no single address received the questionnaire a second time). There were 20,701 responses. Of those completing the questionnaire, 7 percent were professors, 24 percent were undergraduates, 11 percent were graduate students, 16 percent were K-12 students, 5 percent were continuing education continuing education: see adult education.
or adult education
Any form of learning provided for adults. In the U.S. the University of Wisconsin was the first academic institution to offer such programs (1904). students, 4 percent were K-12 teachers, and 25 percent placed themselves in the "other" category.
Clearly, the PDL is finding substantial usage in educational settings as well as by the public outside the classroom. The majority (66 percent) of respondents said that they were using the PDL for the first time, and 11 percent said they used the PDL once a week or more often. The PDL (and other DLs) must consider serving first-time or casual users for the foreseeable future. Of those responding, 54 percent said they were using the PDL from home, 16 percent at school, 14 percent at the office, 3 percent from a library, 3 percent from other places, and 9 percent did not respond. The significant home access data has interface design and system performance implications for digital librarians since home infrastructure support will tend to lag behind institutional infrastructure (e.g., bandwidth, latest client software). It is interesting to note that 37 percent of the respondents said they were using the PDL for personal interest, followed by 23 percent for research, 21 percent for homework, 9 percent for class work, and 9 percent had no answer. In many ways, the PDL is used like a public library in this regard. Of those responding, 36 percent reported that they learned about the PDL from a search engine and 27 percent followed a link from another site. These huge numbers of visits based on searches or links distinguish the PDL from physical libraries. Another 13 percent learned about the PDL from a teacher, 7 percent from a friend, 3 percent from a publication, 6 percent from other sources, and 9 percent provided no answer. These data represent a large sample of DL users and bear reflection as digital librarians develop and upgrade their systems (see Marchionini, Scaife, & Crane, 2000, for the details of this survey).
Document analysis uses the content of a variety of objects to understand goals, outcomes, and processes. In another venue, we used document analysis extensively in assessing user needs for the Library of Congress Digital Library Program (Marchionini, Plaisant, & Komlodi, in press). In the Perseus evaluation, we first focused attention on products produced by the project such as documentation and project reports. Later, with funding from FIPSE, we focused on instructional documents such as syllabi, assignments, and instructional materials such as structured paths through the materials. Other documents analyzed include: the number and range of research papers and conference presentations, link patterns (e.g., WWW links [citations] to the Perseus home page), electronic list messages (number of messages, who participates, and content categories), and original materials contributed by scholars (the Stoa).
Learning analysis was the fourth general class of methodology defined in the evaluation plan and included baseline data such as student reading and translating rates, assignments and syllabi characteristics, and estimates of student performance. Grades and other instructor assessments and structured self-assessments were also proposed. In hindsight, we were able to obtain far less of this type of data than we hoped. Access to student grades is problematic and gathering representative samples of skill levels was logistically impossible. Bill McGrath reported at the Perseus evaluation meeting in 1996 on his systematic analysis of student course evaluations A course evaluation is a paper or electronic questionnaire, which requires a written or selected response answer to a series of questions in order to evaluate the instruction of a given course. for his classes over several years while Perseus was introduced and used. In this case, the evaluation averages went down. Student interview data we collected with McGrath's classes indicated that students very much enjoyed the lively lectures and discussions, were attracted to his courses for this reason, and were somewhat resentful re·sent·ful
Full of, characterized by, or inclined to feel indignant ill will.
re·sentful·ly adv. of time taken away from class discussions as electronic resources were used to augment the lectures. Although student course evaluations are attractive potential sources of evaluation data, the many factors that go into final evaluations and the logistical problems of obtaining access make them less useful in practice. It is important that instructors be prepared to accept negative effects (course evaluations being only one indicator) as well as positive effects when they make significant changes to their courses and teaching styles.
Based on the evaluation plan, we conducted site visits to ten universities from 1989 to 1993 and produced reports for each site visit at the end of each year. In one case, we were able to conduct a controlled comparison of four sections of a large class in which two sections used Perseus and two did not. Other data-collection activities were undertaken opportunistically. For example, Perseus was used in the Fogg Museum exhibit as an adjunct to a classics course, and several Perseus workstations were incorporated into the "Greek Miracle" exhibit at the National Museum of Art (NMA NMA Nederlandse Mededingingsautoriteit
NMA National Medical Association
NMA National Mining Association
NMA NetWare Management Agent (Novell)
NMA New Model Army
NMA National Motorists Association
NMA North Mississippi Allstars ). In both cases, patron questionnaire data were collected and interviews were conducted at the NMA. Marchionini and Crane (1994) report the results of the evaluation to that time with emphasis on the comparative study. Neuman (1991); Morrell, Marchionini, and Neuman (1993); and Marchionini, Neuman, and Morrell (1994) report details for different aspects of the evaluation through the first half-decade.
In 1995, funding to evaluate the educational and scholarly productivity aspects of Perseus was obtained from FIPSE. Although the overall evaluation plan continued to serve as a rubric RUBRIC, civil law. The title or inscription of any law or statute, because the copyists formerly drew and painted the title of laws and statutes rubro colore, in red letters. Ayl. Pand. B. 1, t. 8; Diet. do Juris. h.t. , this funding marked an important juncture junc·ture
The point, line, or surface of union of two parts. in the evaluation focus as the main efforts shifted to teaching and scholarship as the primary emphasis, with learning as an indicator of teaching effects. The first three-year cycle focused on building and assessing instructional materials that incorporated Perseus resources, and the second three-year cycle focused on how Perseus influenced scholarship in support of teaching. The first three years essentially extended the original evaluation plan, although with substantially less support, and the second three years extended the evaluation to the research and scholarship goals of Perseus. Annual reports for these evaluations were produced, and several are available in the Perseus DL under the teaching with Perseus division (http://www.perseus.tufts.edu/FIPSE/) (see Marchionini, Scaife, & Crane, 2000, for a recent report).
Many of the detailed results of the evaluation are available in published reports or WWW sites, so the main findings are briefly summarized here. The results are organized into five categories: physical infrastructure; conceptual infrastructure; mechanical advantage; augmentations; and community development/systemic change.
The results from the early years were strongly influenced by technology. In every interview and observation, issues of using the physical components of the PDL arose. Whether working with a standalone stand·a·lone
Self-contained and usually independently operating: a standalone computer terminal. HyperCard application on a dedicated workstation in a lab, or a network sharing the PDL from magnetic or CD-ROM stores, there were recurring re·cur
intr.v. re·curred, re·cur·ring, re·curs
1. To happen, come up, or show up again or repeatedly.
2. To return to one's attention or memory.
3. To return in thought or discourse. problems with hardware and software reliability software reliability - See also formal methods, safety-critical system.
ftp://ftp.sei.cmu.edu/pub/depend-sw. Mailing list: email@example.com. . These problems were organized under a category labeled "physical infrastructure." These problems were due to a variety of factors, including creating and delivering commercial-grade systems with personal computer platforms and development environments; primitive computing support for faculty offices and laboratories in most universities, especially for the humanities; lack of technical support and training staff; low levels of computer literacy Understanding computers and related systems. It includes a working vocabulary of computer and information system components, the fundamental principles of computer processing and a perspective for how non-technical people interact with technical people. on the part of humanities faculty and students; complex interfaces due to wide ranging content and tools; and poor mass delivery technologies (CD-ROM and networks). In the mid-1990s, moving Perseus to the WWW and increasing experience with digital technology by students, faculty, and universities substantially reduced the number of complaints and comments related to physical infrastructure. Although the interface is still complex due to the variety of material and tools for search and analysis, Perseus is generally accessible twenty-four hours a day, seven days a week to anyone in the world who has an Internet connection. Thus, substantial progress has been realized toward the general mission of providing widespread access to humanities source materials. In hindsight, it is important to note that, if the evaluation of the PDL had ceased after the first four years, the many problems related to physical infrastructure would have likely dominated the results and stymied continued development and decision making. This point again illustrates the importance of taking a longitudinal approach to evaluation.
Along with the need for physical infrastructure, the evaluation quickly made it apparent that there is little guidance and support for using DLs. Teachers do not know how to integrate these resources into their instruction; students do not know how to best use these resources when guided by assignments, let alone direct their own learning with digital resources; scholars do not know how to leverage DLs and incorporate their scholarship into them; and organizations such as universities, museums, and publishers do not know how to value reward, and adopt DLs and the people who work with them. The FIPSE grants aimed to address some of these issues, and we have some good examples of instructors who are integrating their research and teaching through the DL. For example, early in the project, Neel Smith guided students through data collection activities with geospatial data in Perseus to discover new relationships in the altitudes of cities that minted coins in ancient Greece The term ancient Greece refers to the periods of Greek history in Classical Antiquity, lasting ca. 750 BC (the archaic period) to 146 BC (the Roman conquest). It is generally considered to be the seminal culture which provided the foundation of Western Civilization. . Today there are several new examples at the Stoa site (www.stoa.org). For example, Nick Cahill is using his research on Olynthus as part of his course materials, supplementing his scholarly book with online materials, and Christopher Blackwell and a group of collaborators are creating online resources and a public forum related to Athenian democracy
A class of anticipated results showed up repeatedly over the years. The PDL provides people with more information more quickly than otherwise possible. These mechanical advantages provided by the digital medium were evidenced in several ways. First and foremost, access to large amounts of information is available with a few mouse clicks rather than grabbing a book (or, as many students pointed out, walking to the library and finding the book) and finding information manually. Such access is especially important in smaller schools where the library collection does not include broad ranges of texts nor multiple copies of common texts; where slide collections are not extensive and may be highly restricted to faculty and graduate students; and where the artifact A distortion in an image or sound caused by a limitation or malfunction in the hardware or software. Artifacts may or may not be easily detectable. Under intense inspection, one might find artifacts all the time, but a few pixels out of balance or a few milliseconds of abnormal sound collection does not have many examples of vases, sculptures, or other objects. In the WWW environment, this applies beyond the PDL as links in and out lead to much broader arrays of resources. This access is even more crucial from homes; referrer logs show many distance education courses that link to the PDL. Second, selective access is improved and faster for electronic materials. Students were able to do word lookups in the Greek lexicon faster in their translation courses, although some students did express concern that the "ease" of lookup A data search performed within a predefined table of values (array, matrix, etc.) or within a data file. might diminish their translation skills. More common examples were the advantages of doing word searches in Greek or English, which facilitated finding relevant passages when writing essays or constructing arguments.
The results of the comparative study were statistically significant: students found more unique citations when using Perseus than not using it. However, there was no significant improvement in the overall quality of the resulting essays. Third, instructors are able to present more varied examples in class and use these examples in a more facile (language) Facile - A concurrent extension of ML from ECRC.
["Facile: A Symmetric Integration of Concurrent and Functional Programming", A. Giacalone et al, Intl J Parallel Prog 18(2):121-160, Apr 1989]. manner. The range of images and texts in the PDL exceed what might be shown with slides, and moving between images, texts, and maps is easier with a computer than multiple slide projectors, video recorders See DVR, DVD-R and DVD drives. , and overhead transparencies. It is important to qualify this with the observation that setting up and using a computer and projector and mastering the PDL interface is a requisite for such usage that has become less onerous today than it was ten years ago. Fourth, instructors can create directed paths through the materials rather easily. The early versions of Perseus included a path tool that allowed users to record selected portions of their traversal of the database. In the WWW environment, instructors can easily provide sets of URLs interspersed with commentary or questions as part of student assignments. Likewise, students can easily create their own paths/electronic presentations or add URLs in their word-processed papers.
In all these cases, it is important to note that mechanical advantage alone is not sufficient to improve learning or critical thinking. In fact, mechanical advantage raises many issues about learning in electronic environments. What to do with the time that might be saved? How to deal with possible information overload A symptom of the high-tech age, which is too much information for one human being to absorb in an expanding world of people and technology. It comes from all sources including TV, newspapers, magazines as well as wanted and unwanted regular mail, e-mail and faxes. ? How to integrate results from multiple data sources such as texts and vases? Students and instructors sometimes worried about how easy it is to focus on searching and examining results rather than the more challenging activities of reflecting on meaning and creating one's own interpretation of the evidence. As Tom Martin, a Perseus advisor and early user, noted in an interview: "Collecting data comes more easily than interpreting it." Clearly, broad fast access to source materials has been made possible by the PDL, thus achieving one of the guiding missions of the project. The evaluation results answer the questions of what people do with this access and how they manage the new challenges brought by such access. Many examples of exasperation Exasperation
See also Frustration, Futility.
Marine corps sergeant exasperated by Gomer’s ceaseless stupidity. [TV: “Gomer Pyle, U.S.M.C. and rebellion were found--e.g., students who strongly preferred reading assigned secondary works and writing essays rather than conducting investigations in masses of data to discover relationships and make interpretations that might be presented as Perseus paths or Web pages. On the other hand, other students reported being inspired by the self-directed exploration and multiple media. Clearly, individual differences, such as motivation, learning style preferences, and domain knowledge as well as the classroom and university setting, influenced these reactions for students as well as instructors. Rather than focusing on these differences, we early on began to look for examples of ways that the PDL empowered new types of learning and teaching, going beyond the amplifications of mechanical advantage to new augmentations made possible by the PDL.
Clark (1983) long ago warned researchers to avoid media comparison studies when assessing educational technology impacts since the many variables cannot be controlled. Kozma (1991) and Salomon (1979) have argued that the symbol systems of different media do in fact strongly influence learning. All agree that it is extremely difficult to do comparative studies of learning effects. As part of the PDL evaluation, several comparative assessments were made, yielding no definitive effects but rather reinforcing the "it depends" conclusion. Moreover, identifying new effects using existing treatments and metrics is unlikely, and the interviews and observations yielded interesting anecdotes which suggested that new kinds of teaching and learning were emerging. Thus, we began to look for specific examples that would demonstrate how the PDL augmented learning and teaching. Four classes of augmentations are briefly summarized here.
First, students who had no Greek language were able to apply the philological phi·lol·o·gy
1. Literary study or classical scholarship.
2. See historical linguistics.
[Middle English philologie, from Latin philologia, love of learning tools in the PDL to investigate the meanings and nuances of Greek words and associated concepts. Different instructors have used variations of this activity, but the main idea is to explore an important cultural concept (e.g., concepts such as wealth and honor) by: (1) first looking for Looking for
In the context of general equities, this describing a buy interest in which a dealer is asked to offer stock, often involving a capital commitment. Antithesis of in touch with. all occurrences of the term in the Greek to English lexicon (a simple search in the lexicon), thus locating all the Greek words that have this term in their definitions, (2) locating all occurrences of those Greek words in the Greek text corpus In linguistics, a corpus (plural corpora) or text corpus is a large and structured set of texts (now usually electronically stored and processed). They are used to do statistical analysis, checking occurrences or validating linguistic rules on a specific universe. (a set of simple searches in the texts), and (3) reading the English translation of the section of text containing the Greek term (users can display Greek or English versions). By doing so, students were able to see that concepts such as "wealth" carried modern connotations (gold, animals, etc.) but also that the "house" in the sense of family and lineage was an important facet of wealth in ancient Greek culture. Such investigations reflect the kind of work scholars do to build interpretations about ancient cultures. These investigations would have been impossible without the electronic corpus of Greek text and translations and the associated indexes and lexicons. In interviews with students, both strongly positive and negative reactions were voiced--the positive centered on the exploratory investigation, the negative on the time-consuming nature of the searches when a treatise A scholarly legal publication containing all the law relating to a particular area, such as Criminal Law or Land-Use Control.
Lawyers commonly use treatises in order to review the law and update their knowledge of pertinent case decisions and statutes. on the topic could have been read more quickly. Another student wrote on the questionnaire: "The amount of information initially overwhelmed o·ver·whelm
tr.v. o·ver·whelmed, o·ver·whelm·ing, o·ver·whelms
1. To surge over and submerge; engulf: waves overwhelming the rocky shoreline.
a. our research group. It took many hours to glean glean
v. gleaned, glean·ing, gleans
To gather grain left behind by reapers.
1. To gather (grain) left behind by reapers.
2. any meaning from the text references Perseus provided. But those many hours would have been many weeks if it was not for Perseus. In the end, Perseus gave us the ability to come up with an intelligent view of what Herodotus thought about freedom." Clearly, the motivations and styles students bring to tasks affect the effects any DL will have.
Second, students were able to leverage the ready access to volumes of text and visuals to make discoveries and amass evidence to support their discoveries--the PDL is a laboratory for humanities research. One student spent the semester studying vase paintings to determine how women were depicted and gathered evidence that mortal women, except deities
1. Subordinate in capacity or function.
2. Obsequious; servile.
3. Useful as a means or an instrument; serving to promote an end. positions to men. Another student discovered anomalies in vase paintings depicting hoplites without sandals but line drawings in books showing them with sandals. A treatment of the veracity veracity (vras´itē),
n of the historical record became a scholarly theme that grew out of simple curiosity about why, on a vase painting, Sciptians wore sandals but heavily armored Greek soldiers did not. This particular student noted that he spent about fifteen hours looking at images before he made his "discovery." Without traveling to many museums to study these vases, neither of these students would have melded visual thinking into their written work. Other students were able to use text tools to do first rate research. One student noticed that Herodotus uses the concept of "catastrophe" as if it were an infectious disease Infectious disease
A pathological condition spread among biological species. Infectious diseases, although varied in their effects, are always associated with viruses, bacteria, fungi, protozoa, multicellular parasites and aberrant proteins known as prions. . She then investigated the usage in Lysias to compare how a historian and an orator ORATOR, practice. A good man, skillful in speaking well, and who employs a perfect eloquence to defend causes either public or private. Dupin, Profession d'Avocat, tom. 1, p. 19..
2. used the concept and discussed the overlaps and distinctions of a "crippling crip·ple
1. A person or animal that is partially disabled or unable to use a limb or limbs: cannot race a horse that is a cripple.
2. A damaged or defective object or device.
tr.v. agent" in human affairs. Students preparing for a summer course trip to Greece used Perseus to prepare tours of specific sites. Each student was responsible for leading the tour of a site and used the site maps, site photographs, as well as background information in Perseus, in preparing the tour. Clearly, not all students make discoveries, and significant portions of time are spent "surfing" for interesting connections but, as these examples pile up, the value of easy access to large volumes of data begins to emerge.
Third, there were instances of teams of students or students and professors collaborating around the PDL as "electronic campfire." In addition to the example of the coinage coinage
Certification of a piece of metal or other material (such as leather or porcelain) by a mark or marks upon it as being of a specific intrinsic or exchange value. Croesus (r. c. and altitude correlation noted earlier, other instances of spontaneous, as well as forced, collaboration were cited. In one case, a graduate student and professor using word analysis tools discovered that the morphological variation Antigone uses to refer to herself is distinctly different from how others in the play refer to themselves. They argued that Euripides used this as a lexical lex·i·cal
1. Of or relating to the vocabulary, words, or morphemes of a language.
2. Of or relating to lexicography or a lexicon.
[lexic(on) + -al1. device to reinforce her alienation from the rest of society. Another professor was elated e·lat·ed
Exultantly proud and joyful.
e·lat by a discussion that took place in class as the group explored the use of terms for freedom in Greek democracy. He noted new insights he had during class (and shared his excitement with the class) in spite of being a seasoned authority on the subject. Several instructors used group projects and "labs" in their courses. Not surprisingly, the results were mixed. One instructor noted that, although in his class Perseus tended to encourage more group work which, in turn, led to more idea sharing and clarity in expressing those ideas, there were also cases of homogenization homogenization (həmŏj'ənəzā`shən), process in which a mixture is made uniform throughout. Generally this procedure involves reducing the size of the particles of one component of the mixture and dispersing them evenly of thinking and abandonment of responsibility. He summed the collaboration nicely as: "You get brand new highs and brand new lows." The message here regarding a wider range of diversity in behavior has important implications for instruction, DLs in general, and evaluation.
Fourth, the PDL provides the material and tools for new forms of creative expression. In the early days, students used the path tool to create paths through the corpus that represented their interpretation of assignments or ideas. In the WWW environment, students create Web pages that integrate PDL materials or word-processed documents that have images, texts, linguistic analysis results, and live links to the primary materials that support their arguments. Instructors increasingly require students to create such expressions rather than traditional essays. The PDL has become an especially useful resource for humanities scholars as they do research and incorporate their results into instruction. Scholars investigating word senses and uses as part of their translations and interpretations of Greek texts have used the philological analysis tools in Perseus. To explicitly support scholars in many classics subfields, the Stoa consortium was founded in 1997 to create a venue for scholarly research and instructional support. Stoa (www.stoa.org) provides tools and advice for scholars creating electronic documents, develops standards for tagging and displaying these products, and offers an electronic publishing An umbrella term for non-paper publishing, which includes publishing online or on media such as CDs and DVDs. platform for sharing their work and eliciting scholarly feedback. To date, thirteen ongoing research projects are included in the Stoa, and many of these projects incorporate PDL materials and serve to extend the PDL.
COMMUNITY DEVELOPMENT/SYSTEMIC CHANGE
Perhaps the most important long-term developments are changes at organizational levels, such as departments and schools, and the emergence of a community of practice that leverages and advances the PDL. The original project depended on advisory committees that served as liaisons to the larger technical and educational communities. The team also gave talks at regional and national conferences and published papers in journals to inform the broader community about PDL and generate interest and reaction. In addition, the evaluation team conducted site visits to a number of universities which, in addition to the main objective of gathering evidence of use, caused local self-reflection on PDL practice. Instructors and students who participated in interviews were surely cognizant of what they said and likely reflected afterward af·ter·ward also af·ter·wards
At a later time; subsequently.
Adv. 1. afterward - happening at a time subsequent to a reference time; "he apologized subsequently"; "he's going to the store but he'll be back here on the interview and the PDL experience. In addition, member checks (asking interviewees to review the written summaries and interpretations of their comments) facilitated reflections days or weeks later as participants reviewed the summaries. All these efforts served to alert classicists and instructional technologists to the goals and progress of the PDL.
The first tangible example of system change outside of the project staff settings was at Ball State University (BSU BSU Ball State University
BSU Boise State University
BSU Black Student Union
BSU Bemidji State University
BSU Bowie State University (Bowie, Maryland)
BSU Baptist Student Union (college religious organization) ) in the mid-1990s. At site visits, we noted that four professors were using Perseus in their courses, and the department had committed the bulk of its materials and equipment resources to acquiring equipment to deliver the PDL. The faculty leveraged PDL use within the university to garner support for technical innovations that were strongly encouraged by the administration (e.g., faculty served on university-wide committees devoted to educational technology and were profiled in campus publications). In interviews with classics majors, students noted how they had used PDL in their assignments and projects and expected that they would have access to PDL in their graduate programs at other universities (which at the time was highly unlikely). The faculty discussed ways to build PDL into introductory courses so that students in advanced courses could be expected to take advantage of PDL without special instruction. In effect, the PDL was becoming institutionalized--part of the educational culture--in this department even before the WWW version was widely available.
These developments at BSU signaled new developments in classics departments and the field itself. By the late 1990s, job postings for faculty in several classics departments began to include requirements or preferences for technological skills. The core Perseus faculty were obtaining tenure and promotion at the schools to which they had moved. Yale University Press sold out of the first run of CD-ROMs, and the second edition served as an alternative to the WWW-based PDL. New textbooks began to include supplemental course materials that incorporated Perseus, and some online encyclopedias An online encyclopedia is an encyclopedia which is accessible by computer over the internet. Specific online encyclopedias include:
As the WWW version of the PDL continued to evolve, the evidence of its impact on the field grew as more and more people accessed the corpus. Figure 1 summarizes WWW requests over a four-year period. Note that these numbers represent page requests rather than all transfers on a page (e.g., a page with
five GIFs counts as one request even though the transaction logs contain six http requests). Note that spikes in usage recur during academic periods. In spring 2000, the PDL was responding to as many as 250,000 requests per day. The AltaVista portal listed almost 30,000 links to the PDL home page in mid June 2000 compared to almost 56,000 for the Library of Congress home page. Thus, the transaction log data provide another powerful indication that the PDL has become an important part of the humanities infrastructure.
[Figure 1 ILLUSTRATION OMITTED]
Yet another indication of the influence of the PDL in education and the scholarly community is its expansion from Greek and Roman culture to other humanities areas as holders of important intellectual property are drawn to the technical and editorial expertise that has accrued at the PDL. The issues of editing and managing large corpora corpora
plural form of corpus.
see corpus albicans.
sandy or gritty bodies, found in the pineal body; appear to be of glial or stromal origin; have the structure of of source materials have led to demand for new roles and new skill sets for scholars in the humanities. The need to explicitly address the challenges of training and promoting scholars who demonstrate both domain and technical excellence was made explicit in a recent paper (Crane & Rydberg-Cox, 2000) that called for post doctoral positions in the humanities to support corpus editors.
REFLECTIONS AND RECOMMENDATIONS
The evaluation effort has explicated many of the outcomes and challenges related to the PDL over more than a decade. The results above provide guidance to other DLs and shed light on questions at many levels. Is the impact worth some $10 million of investment? How has the PDL influenced learning? Teaching? Scholarly research? Is the PDL still viable? Sustainable? How does the evaluation inform continued development? The evaluation offers partial answers to all these questions as discussed in the results synthesis above. In this final section, attention is focused on the evaluation process itself and on recommendations for digital library evaluation in other settings. Three main points about DL evaluation with associated corollaries are offered:
1. Evaluation efforts must explicate goals on a continuum ranging from evaluation research to product/system testing.
At the research end of the spectrum, the goals are related to understanding complex phenomena through inference and chains of evidence. At the product/system-testing end, the goals are related to direct measurement of well-specified criteria that inform practical decision making. Most evaluation efforts in academic settings fall somewhere in between, using direct measurement and inference chains to build arguments and cases that inform decision making and continued development. In the PDL evaluation plan, formative and summative Adj. 1. summative - of or relating to a summation or produced by summation
additive - characterized or produced by addition; "an additive process" components were originally specified to distinguish this continuum.(9) What is important for evaluation research is to gather and integrate as many specific measures as feasible without depending too heavily on any single measure. Metrics such as number of HTTP requests, number of objects digitized, response rates, server down (or busy) time, interface feature sets, error rates, number of abandonments,(10) satisfaction ratings, interview comments, e-list traffic, cost per request,(11) and interesting anecdotes provide important glimpses into the DL phenomenon and context but individually do not provide a full view. It is surely possible and necessary to measure these results, but they are only threads in the more substantive evaluation questions asked in a research vein. Evaluation research that incorporates multiple data threads yields a complex fabric of effects that itself changes shape and meaning depending on the light and angle of view. Clearly, explicating evaluation goals on the research side of the continuum implies more cost and time commitments--factors that must be taken into account as evaluation is planned. If limited resources are available, focusing on a small set of well-defined system effectiveness questions may be prudent. However, because DLs are emergent systems, more ambitious evaluation efforts that gather baseline data and track changes over time are encouraged because they will more strongly benefit the DL community in the long run.
2. Digital libraries are emergent complex systems.
Digital libraries meld electronic tools and procedures with the entire range of forms of human expression. This includes new forms of expression made possible by the technology associated with DLs. The resulting complexity may yield effects that are greater than the sum of the parts--i.e., emergent properties (Kauffman,1995). The PDL offers some hints of this in the ways that students are empowered to do research and investigations traditionally reserved for graduate students, professors, and other scholars; in the implications of the PDL as a humanities laboratory; in the incorporation of PDL sources into the basic infrastructure of classical studies; and in the need for new skill sets for humanities scholars who leverage technical tools to create new interpretations and expressions. If DLs are emergent phenomena, DL evaluation must surely be designed to seek unexpected outcomes. Two characteristics that support such an evaluation are longitudinality and flexibility.
The original evaluation plan aimed to address a number of general questions over time by using a variety of data collection and analysis techniques. The plan was designed to be flexible in that techniques and questions could be adapted as the PDL itself and the technological and cultural contexts changed. This flexibility and attention to the interactions between the PDL and its environment is a defining characteristic of what might be termed "emergent evaluation." Emergent phenomena are driven by a small set of rules that control how systems interact with the environment (Clark, 1997). In the case of evaluation, the rules are determined by a high-level mission and data collection techniques. For the PDL, evaluation of the emergent phenomena of electronic resources in the humanities is controlled by the mission of broad access to source materials and a set of techniques that are adapted to the environmental conditions in which the mission operates. Over a dozen years, the environments included many physical sites (various instructional and research settings), a range of physical infrastructure developments (from single stand-alone PCs A desktop or laptop computer that is not permanently connected to a local area network (LAN) or a wide area network (WAN). Throughout the 1990s, millions of stand-alone PCs were networked in offices, but it is no longer uncommon to find computers networked in the home so that family in a department office running primitive hypertext software to high-end workstations in dorms, homes, and offices linked through the WWW delivering software and content supported by 24/7 campus system administrators), and a growing range of conceptual infrastructure (from novice students and teachers with no experience using technology in learning and teaching to highly computer literate computer literacy
The ability to operate a computer and to understand the language used in working with a specific system or systems.
computer literate adj. students and faculty). If the evaluation had concluded after five years, the effects of physical infrastructure would have dominated the results. The longer view illustrates some of the more substantive effects of the PDL as a critical infrastructure for the humanities and an important focal point focal point
See focus. for computing in the humanities.
A rationalistic approach to evaluation would compare the effects of a DL with the effects of a physical library on library effectiveness metrics. Various components effects (e.g., technical, content, and individual user) would ideally be separated out and weighted to produce some predictive model that explains performance and informs subsequent design. Unfortunately, such an approach oversimplifies both the components and metrics and, more importantly, does not take into account the higher-order interactions that emerge when dynamic systems operate in the real world. Positive outcomes on one dimension lead to unpredictable side effects Side effects
Effects of a proposed project on other parts of the firm. on another dimension. In contrast, the longitudinal and multifaceted approach taken in the PDL evaluation was able to tolerate contradictions and side effects by looking at long-term effects and juxtapositioning data from multiple sources. The many specific questions under the four classes of main questions served to guide questionnaire and structured protocol developments and guide observations but, because the intention was to develop a high-level road map rather than a detailed blueprint, a richer process that captured some of the emergent properties of the PDL was possible.
3. Integrate statistical data and narratives to assess impact as well as performance and usage
Like circulation and holdings data in physical libraries, transaction log summaries and other performance data demonstrate operational effects of the library but do not explain how this usage impacts stakeholders Stakeholders
All parties that have an interest, financial or otherwise, in a firm-stockholders, creditors, bondholders, employees, customers, management, the community, and the government. . Marchionini has argued (Marchionini, 1995; Marchionini, Plaisant, & Komlodi, in press) that impacts change over time and vary by stakeholder stakeholder n. a person having in his/her possession (holding) money or property in which he/she has no interest, right or title, awaiting the outcome of a dispute between two or more claimants to the money or property. (e.g., individuals, groups, organizations, society). Operational data are powerful components in a chain of inferences that address impact, but the PDL evaluation illustrates the value of anecdotes and "stories" that illustrate new effects--i.e., how DLs augment existing capabilities with new ones. These augmentations garner public support for a DL and should not be underestimated in assessing impact.
In evaluating DLs, it is important to consider the changes that DLs bring and only some of these are explainable through deltas in statistical data. Some of these changes are positive, but others will be controversial. Because human attention is a finite resource, new or additional capabilities can displace dis·place
tr.v. dis·placed, dis·plac·ing, dis·plac·es
1. To move or shift from the usual place or position, especially to force to leave a homeland: or reprioritize existing capabilities. Seeking and documenting these changes can be uncomfortable, especially when the evaluation funding is tied to the DL. Integrating multiple views is more naturally done with narratives rather than summary statistics, and integrating these forms of evidence can aid in assessing complex change.
Similarly, it is evident that DLs will lead to more diversity. Beyond the obvious broader access by global populations of users, the ranges of materials and new tools for access and use lead to new "highs and lows" of human application. Evaluation that treats these ranges may not make funding bodies A funding body is an organisation that provides funds in the form of research grants or scholarships. Research Councils
Research Councils are funding bodies that are government-funded agencies engaged in the support of research in different disciplines and or traditional user communities happy, but they will prepare the way for the changes that are inevitable as new information phenomena take their place in the realms of education and scholarship. Narrative explanations are crucial here as well.
The PDL evaluation reveals some of the complex interactions among information resources, stakeholders, and technology. Several observations about the success of the PDL may inform the development and evaluation of other DLs. Digital library success is aided by:
* clear missions;
* strong leadership and a strong talent pool;
* good technical vision and decisions;
* quality content and data management;
* giving users multiple access alternatives; and
* ongoing evaluation effort.
As we worked to understand the mission of Perseus in the early days, many metaphors were used by the staff to describe the project vision. One constant was the mission to maximize access to source materials. This mission immediately led to many benefits that garnered support and understanding. These include: allow people to make their own interpretations rather than learning accepted dogma DOGMA, civil law. This word is used in the first chapter, first section, of the second Novel, and signifies an ordinance of the senate. See also Dig. 27, 1, 6. (facilitating critical thinking); critical mass of content facilitates new discoveries; extending appreciation of classical culture makes classics more viable and sustainable in university curricula. This clear and populist mission served the PDL well over the years, and other DLs can benefit from a clear and crisply articulated mission statement.
Leadership is important to organizational success and, although DLs may exist in virtual space, the resources behind the scenes are real and must be assembled, inspired, and managed. The PDL has had a single chief and highly stable steering group throughout its history. Additionally, the talent pool offered by several major universities supplied the manpower to build the PDL. Continuous leadership works with a clear mission to attract and inspire such talent. Knowing the needs of stakeholders is important to leadership--this includes potential funders and discipline leaders as well as staff and end users. Tenacity and commitment lead to ongoing dissemination and evangelizing that garner support and usage. Although this is hardly surprising in any organizational setting, it is worth pointing out that it clearly extends to DLs in cyberspace Coined by William Gibson in his 1984 novel "Neuromancer," it is a futuristic computer network that people use by plugging their minds into it! The term now refers to the Internet or to the online or digital world in general. See Internet and virtual reality. Contrast with meatspace. .
The PDL benefited from staff and advisory boards that recognized important trajectories in technical development. Good decisions about storage and dissemination (e.g., CD-ROM for standalone and LAN (Local Area Network) A communications network that serves users within a confined geographical area. The "clients" are the user's workstations typically running Windows, although Mac and Linux clients are also used. use, early adoption of WWW), system architecture (e.g., object-oriented design Transforming an object-oriented model into the specifications required to create the system. Moving from object-oriented analysis to object-oriented design is accomplished by expanding the model into more and more detail. , database lookups rather than hard-coded hyperlinks, and current emphasis on open source tools), and multi-platform delivery (e.g., stand-alone and WWW) all allowed the PDL to evolve while technology changed dramatically. It is easy to say that good technical decisions are required for DL success but harder to put into practice. Excellent advisory groups that represent different points of view and are willing to give regular attention to progress are difficult to assemble. They should not be political bodies--once they are assembled, the DL staff must be prepared to take full advantage of their advice.
Likewise, decisions about content and data models are crucial to success. The PDL collection development policies were informed by an advisory board and are also opportunistic. It was aided in some sense by a finite set In mathematics, a set is called finite if there is a bijection between the set and some set of the form where n is a natural number. (The value n = 0 is allowed; that is, the empty set is finite.) An infinite set is a set which is not finite. of ancient Greek texts, but there were many arguments over which texts to include and which museums and sites to approach for images. Good decisions about data models (e.g., catalog card metadata for art objects), data descriptions (e.g., adoption of SGML), and a general focus on content rather than technology may have been aided by the fact that classicists understand the importance of persistence of data and were prepared to plan for a DL that would outlive out·live
tr.v. out·lived, out·liv·ing, out·lives
1. To live longer than: She outlived her son.
2. the latest technical solution.
The PDL grew out of the early hypertext research of the 1980s and thus was rooted in the notion of giving users control through multiple links and access points. The philological tradition of concordances concordances,
n.pl 1. items that are in harmony.
2. homeopathic medicines with affinity to one another and therefore can be used serially during the sequence of treating an illness. This interaction was initially noted by Boenninghausen. and systematic word searches brought advanced search capabilities to the corpus from the earliest days. This combination of support for browsing and analytical search supports diverse usage and is an important lesson for other DLs. Giving people control over how they access and use the DL satisfies a broader range of users and gives rise to wider ranges of applications.
Finally, an ongoing evaluation effort serves multiple purposes. Evaluation serves a political/administrative role by providing the reports and data upon which decisions about funding and development may be based. Evaluation results also inform the ongoing development of the DL both technically and conceptually. Evaluation activities also serve to involve staff and users in the work of the DL at reflective levels that may improve usage and support. Evaluation serves to document the evolution of a particular DL. Most importantly, evaluation with a research focus helps to explain the effects of a specific DL and relate it to the larger issues of DL evolution and impact.
Perseus has always been a library. Although it was not fashionable (or fundable?) as a digital library in the mid to late 1980s when libraries were written off as anachronisms and library and information science programs were closing, the aim of making source materials widely available, the emphasis on self-directed learning, the organizational schemes applied for preserving and ensuring scholarly access, and the access and analysis tools created all reflect information science theory and practice. Perhaps better than the other answers about hypertextual systems, digital objects, interactive curricula, and communities of practice, the best answer to Ehrmann's question is that Perseus is a digital library. It has continued to evolve and stands as a significant city in a cyberspace that is now being defined by a vast network of linked digital libraries with tangible connections to the physical artifacts that make up our world.
Over the years, it has been my great fortune to be able to draw upon a diverse range of talented people who participated in the evaluation. Kenny Morrell was an original member of the Perseus team and helped shape the original evaluation plan and conduct the early studies. Delia Neuman was my original research partner and worked with Morrell and me to develop the plan and was instrumental in collecting data in the early years and creating analysis strategies. Steve Erhmann, who served as program director at Annenberg CPB CPB
see cardiopulmonary bypass.
CPB Cardiopulmonary bypass. See Port-Access cardiopulmonary bypass. in the early days, prodded us with excellent questions and supported us as we elaborated on our approach to longitudinal formative evaluation Formative evaluation is a type of evaluation which has the purpose of improving programmes. It goes under other names such as developmental evaluation and implementation evaluation. . Peter Evans Peter Evans may refer to:
In the investing world, a totally new way of doing things that has a huge effect on business.
The word "paradigm" is defined as a pattern or model, and it has been used in science to refer to a theoretical framework. for teaching. Partial funding for the evaluation was provided by the Annenberg/CPB Project and the Fund for Improvement of Post Secondary Education.
(1) A path mechanism that teachers and students could use to record sequences through the corpus was provided as a tool.
(2) See Daniels & Chavez (1999) for instructions for contributing to the Perseus Digital Library collection.
(3) The current PDL offers CD-ROM and WWW versions. Because some museums restricted image use to CD-ROM, the image collections differ according to intellectual property rights agreements. Likewise, full Greek lexicon versions and some materials beyond the original Greek culture corpus are only available in the WWW version. Additionally, the technical capabilities differ slightly as WWW display and transfer capabilities differ from what is possible with locally mounted CD-ROM. See http://www.perseus.tufts.edu/ order.html for a comparison chart of the two instantiations of the PDL.
(4) Bloom, in his taxonomy of educational objectives The Taxonomy of Educational Objectives, often called Bloom's Taxonomy, is a classification of the different objectives and skills that educators set for students (learning objectives). , poses evaluation as the highest of six cognitive goals for learners (Bloom, 1956).
(5) An example of a comparative question in the learner group is 4.2. In different courses with similar objectives, are there differences in the amounts and kinds of achievement for courses supported by Perseus and those not supported by the system?
(6) An example of a usage question in the instructor group is 2.1. What tactics, strategies, and patterns do instructors develop with Perseus in connection with their teaching? Research?
(7) An example of a system question is 3.4. How are student-defined paths used? Shared?
(8) An example of a content question is 3.2. How well can Perseus help the student or scholar clearly delineate fact from scholarly conjecture CONJECTURE. Conjectures are ideas or notions founded on probabilities without any demonstration of their truth. Mascardus has defined conjecture: "rationable vestigium latentis veritatis, unde nascitur opinio sapientis;" or a slight degree of credence arising from evidence too weak or too ?
(9) Flagg (1990) provides a good set of formative evaluation case studies related to educational technology projects.
(10) Hert and Marchionini (1997) report 50% abandonments in 1996 from non .gov, .com. and .edu domains at the BLS See Bureau of Labor Statistics. Web site but half as many abandonments a year later. Thus, these data must be considered over time.
(11) It costs the government $14 if a person requests federal tax forms by phone, $7 by mail, and $3.50 by walking into a post office to pick them up in person. Online requests for those forms cost the government 13 cents (Trimble, 2000). The cost of the citizen's time to obtain the form is an additional factor beyond the government savings.
Bednarz, D. (1985). Quantity and quality in evaluation research: A divergent view. Evaluation and Program Planning, 8(4), 289-306.
Bloom, B., & Krathwahl D. (Ed.). (1956). Taxonomy of educational objectives: The classification of educational goals, by a committee of college and university examiners: Handbook I: Cognitive domain cognitive domain,
n area of study that deals with the processes and measurable results of study, as well as the practical ability to apply intelligence. . New York New York, state, United States
New York, Middle Atlantic state of the United States. It is bordered by Vermont, Massachusetts, Connecticut, and the Atlantic Ocean (E), New Jersey and Pennsylvania (S), Lakes Erie and Ontario and the Canadian province of : Longmans Green.
Campbell, D. (1979). "Degrees of freedom" and the case study. In T. Cook & C. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research (pp. 49-67). Beverly Hills Beverly Hills, city (1990 pop. 31,971), Los Angeles co., S Calif., completely surrounded by the city of Los Angeles; inc. 1914. The largely residential city is home to many motion-picture and television personalities. , CA: Sage.
Chavez, R. (2000). Generating and reintegrating geospatial data. In R. Furuta (Ed.), Proceedings of the Fifth ACM (Association for Computing Machinery, New York, www.acm.org) A membership organization founded in 1947 dedicated to advancing the arts and sciences of information processing. In addition to awards and publications, ACM also maintains special interest groups (SIGs) in the computer field. Conference on Digital Libraries (San Antonio San Antonio (săn ăntō`nēō, əntōn`), city (1990 pop. 935,933), seat of Bexar co., S central Tex., at the source of the San Antonio River; inc. 1837. , TX, June 2-7, 2000) (pp. 250-251). New York: ACM Press.
Clark, A. (1997). Being there: Putting brain, body, and the world together again. Cambridge, MA: MIT MIT - Massachusetts Institute of Technology Press.
Clark, R. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459.
Cook, T., & Reichardt, C. (Eds.). (1979). Qualitative and quantitative methods in evaluation research. Beverly Hills, CA: Sage.
Crane, G. (1988). Redefining the book: Some preliminary problems. Academic Computing, 2(5), 6-11, 36-41.
Crane, G. (Ed.). (n.d.-a). The Perseus Project. Retrieved August 29, 2000 from the World Wide Web: http://www.perseus.tufts.edu.
Crane, G. (Ed.). (n.d.-b). Information about Perseus. The Perseus Project. Retrieved August 29, 2000 from the World Wide Web: http://www.perseus.tufts.edu/PerseusInfo.html.
Crane, G., & Mylonas, E. (1988). The Perseus Project: An interactive curriculum on classical Greek civilization. Educational Technology, 28(11), 25-32.
Crane, G., & Rydberg-Cox, J. (2000). New technology and new roles: The need for "corpus editors." In R. Furuta (Ed.), Proceedings of the Fifth ACM Conference on Digital Libraries (San Antonio, TX, June 2-7, 2000) (pp. 252-253). New York: ACM Press.
Daniels, M., & Chavez, R. (1999). A guide to photographing architecture, monuments, sites, and topography. Retrieved August 29, 2000 from the World Wide Web: http://www.stoa.org/ guides/sitestds.shtml.
Ericsson, K., & Simon, H. (1984). Protocol analysis: Verbal reports as data. Cambridge: MIT Press.
Evans, P. (1993). The enabling and disabling dis·a·ble
tr.v. dis·a·bled, dis·a·bling, dis·a·bles
1. To deprive of capability or effectiveness, especially to impair the physical abilities of.
2. Law To render legally disqualified. effects of a hypermedia hypermedia: see hypertext.
The use of hyperlinks, regular text, graphics, audio and video to provide an interactive, multimedia presentation. All the various elements are linked, enabling the user to move from one to another. information environment on information seeking Information seeking is the process or activity of attempting to obtain information in both human and technological contexts. Information seeking is related to, but yet different from, information retrieval (IR). and use in an undergraduate course. Doctoral dissertation, University of Maryland University of Maryland can refer to:
Flagg, B. (1990). Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum Associates.
Griffiths, J-M J-M Jelinski-Moranda (reliability model) ., & King, D. (1991). A manual for the evaluation of information centers and services (AGARDograph No. 310). New York: American Institute of Aeronautics and Astronautics The American Institute of Aeronautics and Astronautics (AIAA) is the professional society for the field of aerospace engineering. The AIAA was founded in 1963 from the merger of four earlier societies: the American Rocket Society (ARS), founded in 1930 as the Technical Information Service.
Harter, S., & Hert, C. (1997). Evaluation of information retrieval systems: Approaches, issues, and methods. Annual Review of Information Science and Technology, 32, 3-94.
Hert, C., & Marchionini, G. (1997). Seeking statistical information in federal Websites: Users, tasks, strategies, and design recommendations. Final report to the Bureau of Labor Statistics, July 1997. Retrieved August 29, 2000 from the World Wide Web: http://ils.unc.edu/ ~march/blsreport/mainbls.html.
Kauffman, S. (1995). At home in the universe: The search for laws of self-organization and complexity. New York: Oxford University Press.
Kozma, R. (1991). Learning with media. Review of Educational Research, 61(2), 179-211.
Marchionini, G. (1995). The costs of educational technology: A framework for assessing change. In H. Maurer (Ed.), Educational multimedia and hypermedia, 1995 (Proceedings of ED-MEDIA '95--World Conference on Educational Multimedia and Hypermedia, Graz, Austria, June 20, 1995) (pp. 33-38). Charlottesville, VA: Association for the Advancement of Computing in Education. Also available on the World Wide Web at: http://ils.unc.edu/~march/costet/costet.html.
Marchionini, G., & Crane, G. (1994). Evaluating hypermedia and learning: Methods and results from the Perseus Project. ACM Transactions on Information Systems, 12(1), 5-34.
Marchionini, G., & Fox, E. (1999). Progress toward digital libraries: Augmentation AUGMENTATION, old English law. The name of a court erected by Henry VIII., which was invested with the power of determining suits and controversies relating to monasteries and abbey lands. through integration. Information Processing & Management, 35(3), 219-225.
Marchionini, G.; Neuman, D.; & Morrell, K. (1994). Directed and undirected tasks in hypermedia: Is variety the spice of learning? In T. Ottman & I. Tomek (Eds.), Educational multimedia and hypermedia, 1994 (Proceedings of ED-MEDIA '94--World Conference on Educational Multimedia and Hypermedia, Vancouver, BC, June 25-30, 1994) (pp. 373-378). Charlottesville, VA: Association for the Advancement of Computing in Education.
Marchionini, G.; Neuman, D.; & Morrell, K. (1989). Perseus evaluation plan. Unpublished Perseus Project Working Paper Number 5, Dec. 1989.
Marchionini, G.; Plaisant, C.; & Komlodi, A. (forthcoming). The people in digital libraries: Multifaceted approaches to assessing needs and impact. In A. Bishop, B. Buttenfield, & N. VanHouse (Eds.), Digital library use: Social practice in design and evaluation. Cambridge, MA: MIT Press.
Marchionini, G.; Scaife, R.; & Crane, G. (2000). Final evaluation on the Perseus Project publication model, 1997-2000. Retrieved September 12, 2000 from the World Wide Web: http://www.ils.unc.edu/~march/perseus/final_report.pdf.
Morrell, K.; Marchionini, G.; & Neuman, D. (1993). Sailing Perseus: Instructional strategies for hypermedia in the classics. Journal of Educational Multimedia and Hypermedia, 2(4), 337-353.
Mylonas, E. (1987). Using Perseus in a variety of educational settings. (Perseus Working Papers working papers
Legal documents certifying the right to employment of a minor or alien.
Noun 1. working papers , 2), Cambridge, MA, December, 1987.
Neuman, D. (1991). Evaluating evolution: Naturalistic nat·u·ral·is·tic
1. Imitating or producing the effect or appearance of nature.
2. Of or in accordance with the doctrines of naturalism. inquiry and the Perseus Project. Computers and the Humanities, 25(4), 239-246.
Nielsen, J. (1993). Usability engineering Usability engineering is a subset of human factors that is specific to computer science and is concerned with the question of how to design software that is easy to use. It is closely related to the field of human-computer interaction and industrial design. . Boston: Academic Press.
Rossman. G., & Wilson, B. (1985). Numbers and words: Combining quantitative and qualitative methods in a single large-scale evaluation study. Evaluation Review, 9(5), 627-643.
Salomon, G. (1979). Interaction of media, cognition cognition
Act or process of knowing. Cognition includes every mental process that may be described as an experience of knowing (including perceiving, recognizing, conceiving, and reasoning), as distinguished from an experience of feeling or of willing. , and learning: An exploration of how symbolic forms cultivate mental skills and affect knowledge acquisition. San Francisco San Francisco (săn frănsĭs`kō), city (1990 pop. 723,959), coextensive with San Francisco co., W Calif., on the tip of a peninsula between the Pacific Ocean and San Francisco Bay, which are connected by the strait known as the Golden , CA: Jossey-Bass.
Saxton, M. (1997). Reference service evaluation and meta-analysis: Findings and methodological issues. Library Quarterly, 67(3), 267-289.
Suchman, E. (1967). Evaluative research: Principles and practice in public service and social action programs. New York: Russell Sage Russell Sage (4 August 1816 - 22 July 1906) was a financier and politician from New York.
Sage was born at Verona in Oneida County, New York. He received a public school education and worked as a farm hand until he was 15, when he became an errand boy in a grocery conducted Foundation.
Timble, P. S. (2000). A question of trust. Federal Computer Week. Retrieved August 29, 2000 from the World Wide Web: http://www.fcw.com/fcw/articles/2000/0522/pol-trust05-22-00.asp.
Voorhees, E., & Harman, D. (2000). Overview of the sixth Text REtrieval Conference (TREC-6). Information Processing & Management, 36(1), 3-35.
White, H., & McCain, K. (1989). Bibliometrics. Annual Review of Information Science and Technology, 24, 119-186.
Williams, D. (Ed.) (1986). Naturalistic evaluation (New directions for program evaluation, No. 30). San Francisco, CA: Jossey-Bass.
GARY MARCHIONINI is the Cary C. Boshamer Professor of Information Science in the School of Information and Library Science at the University of North Carolina at Chapel Hill The University of North Carolina at Chapel Hill is a public, coeducational, research university located in Chapel Hill, North Carolina, United States. Also known as The University of North Carolina, Carolina, North Carolina, or simply UNC . His Ph.D. is from Wayne State University Wayne State University, at Detroit, Mich.; state supported; coeducational; established 1956 as a successor to Wayne Univ. (formed 1934 by a merger of five city colleges). in mathematics education with emphasis on educational computing. His research interests are in information seeking in electronic environments, digital libraries, human-computer interaction Human-computer interaction
An interdisciplinary field focused on the interactions between human users and computer systems, including the user interface and the underlying processes which produce the interactions. , and information technology policy. He has had grants or contracts from the National Science Foundation, U. S. Department of Education, Council on Library Resources, the National Library of Medicine, the Library of Congress, the Kellogg Foundation Kellogg Foundation, philanthropic institution established (1930) at Battle Creek, Mich., by food manufacturer W. K. Kellogg (1860–1951). Kellogg eventually gave the institution a total of $47 million, and by 1990 its endowment had increased to more than $3. , and NASA NASA: see National Aeronautics and Space Administration.
in full National Aeronautics and Space Administration
Independent U.S. , among others. He was the conference chair for the ACM Digital Library `96 Conference and serves on the editorial board of ten scholarly journals. He has published more than seventy articles, chapters, and conference; papers in the information science, computer science, and education literatures. He recently founded the Interaction Design Laboratory at UNC.
Gary Marchionini, School of Information and Library Science, CB #3360, 203 Manning Hall, University of North Carolina North Carolina, state in the SE United States. It is bordered by the Atlantic Ocean (E), South Carolina and Georgia (S), Tennessee (W), and Virginia (N). Facts and Figures
Area, 52,586 sq mi (136,198 sq km). Pop. , Chapel Hill, NC 27599-3360