Indexing learning objects: vocabularies and empirical investigation of consistency.

In addition to the LOM standard and instructional design specifications, as well as domain specific indexing vocabularies, a structured indexing vocabulary for the more elementary learning objects is advisable in order to support retrieval tasks of developers. Furthermore, because semantic indexing is seen as a difficult task, three issues concerning consistency in indexing learning objects were empirically investigated: 1) the extent to which different indexers annotate in the same way; 2) the extent to which structure in value lists supports consistent indexing; and 3) different degrees of consistency in annotating various media types and attributes. The results show that a standardized value list does not necessarily lead to a consistent application to learning objects. Differences occur, especially for more abstract attributes and media types. Ontologies can contribute to a higher consistency in indexing and could improve retrieval by making concepts that are more abstract available.

**********

The Internet is becoming an important source of e-learning material, as is evident from the steadily growing number of learning object repositories and an increasing market interest. To fully realize the opportunities that are available for distribution and re-use of these resources, several problems must be resolved. First, there should be agreement about what constitutes a learning object. In addition, a knowledge-rich indexing structure for the more elementary learning objects, such as a paragraph of text or a single image, may help developers of learning material to retrieve these fragments more easily. As Polsani (2003) points out: "It is important that the developers agree on a set of specifications for development of learning objects covering such areas as technology, editorial requirements, and stylistic considerations. A commonly agreed standard will enable genuinely sharable and reusable content objects." Mohan and Greer (2003) essentially stress the same issue: "There is yet no standard way to specify content in a learning object using XML. Moreover, the problem does not end with a set of commonly understood content tags. Content for different domains will need different markup tags." However, even if there is a standard, this does not guarantee that the application of this standard to the same learning objects by different people will yield the same outcomes. If there are differences, this will be detrimental to the effectiveness and efficiency of retrieval for re-use. By combining theoretical and empirical research, this paper addresses these three issues.

The first part is a theoretical discussion about what constitutes a learning object and what kind of annotation structure is required to classify learning objects of an elementary grain size. An annotation structure is defined as a collection of attributes (i.e., data elements) and attribute values. The possible attribute values are defined in an indexing vocabulary (i.e., vocabulary value spaces). Several indexing vocabularies that are available to classify learning objects of an elementary grain size are reviewed. In addition, an example of a knowledge-rich vocabulary, which is based on ontologies for fragments of technical manuals that are used for developing lesson material, is provided. Although a knowledge-rich indexing vocabulary for document fragments can improve retrieval performance and to some extent lead to a better product created from the retrieved material (Kabel, Wielinga, de Hoog & Anjewierden, 2003), there is a downside in the effort needed to annotate material. Automatic indexing with ontologies can be achieved to some degree (Anjewierden & Kabel, 2001), but most semantic tagging still must be done manually. As is stressed by the ARIADNE foundation, one of the practical problems that arises when a metadata system is widely used is indexation (i.e., the creation of the metadata by humans), which should be as easy as possible. However, this does not, by itself, lead to consistency in indexing by different people.

The second part reports on an empirical investigation that addresses the question of whether different indexers are capable of applying a standardized value list consistently. Adding semantic metadata to a learning object by an indexer is based on his or her interpretation of the relation between the available values and the learning object. From this perspective, defining a metadata standard is only half the story. The other half is to make sure that differences between indexers are minimized. Ontologies may be a means to promote consistent indexing because, as opposed to flat lists of values, structured lists of values may make it easier for indexers to index learning objects in a consistent way; the structure leaves less room for different interpretations. To gain insight into the degree of consistency in indexing, three questions are investigated. First, to what extent do different indexers, having the same attributes (i.e., metadata elements) and value lists at their disposal, annotate the same learning objects with the same values? Second, does annotating with structured value lists lead to more correspondence between indexers than flat value lists? Third, is consistency of indexing dependent on media types (e.g., video or text), the nature of the annotation structure, different types of attributes, or a specific domain? Before the experiments are described, the rationale behind developing the annotation schema used in the experiments is explained.

DISCUSSION

Indexing What? Learning Objects of an Elementary Grain Size

In the IEEE's standard for Learning Object Metadata (LOM, 2002), learning objects are defined as "any entity, digital or non digital, that may be used for learning, education or training." Learning objects can vary in grain size from raw media data or fragments to a set of courses that can lead to a certificate. Similarly, in ARIADNE's Educational Metadata Recommendation (EMR, 2002), learning objects are described as pedagogic resources that can vary in grain size from a document to a course.

Although these definitions are adequate to achieve basic interoperability between systems, there has been a need to define the continuum from elementary information fragments to more complex instructional structures more precisely. At one end of this continuum are fragments of an informational nature that contain data, such as facts, to inform. As more instructional ingredients, such as an advance organizer or some questions, are added to a piece of information, its size and structure increase; in parallel, its instructional meaning also increases. This line of thought is followed by Cisco Systems (2001), distinguishing Reusable Information Objects (RIO) from Reusable Learning Objects (RLO). An RIO consists of content, practice, and assessment items; an RLO consists of several RIOs, sandwiched between an overview and a summary.

Wiley (2000) provides a theoretical foundation for designing and sequencing learning objects and scopes the LOM definition of learning objects down to "digital resources that can be re-used to support learning." To optimize reusability, the most elementary learning object should be as small as possible with preservation of at most one concept represented in the object that retains meaning when isolated outside the original context. Of the five learning object types Wiley defines, the most elementary type is an individual digital resource uncombined with any other, for instance, an image that serves as an example. Using this definition, because our interest is in optimizing reusability, we focus on digital learning objects of the most elementary type; we will call these fragments, for example, a short video, an image, or a paragraph of text.

What Structure and Content are Necessary to Annotate Fragments?

In the context of re-use through the Internet, one should consider a wide variety of people who wish to retrieve a wide variety of fragments for many different instructional goals and tasks. This clearly requires a rich and many-faceted annotation facility. A real-life work task, such as developing lesson material, drives one or more information-seeking tasks (i.e., the need for information), which may comprise one or more information retrieval tasks (i.e., consulting a source) (Bystrom & Hansen, 2002). This line of retrieving fragments, followed by developers, can be supported: 1) by providing a rich expression of knowledge in structured value lists, and 2) by representing domain information and information about the material itself, including semantic descriptions of the content and the possible roles of a fragment in instruction. Both can be achieved by using ontologies as indexing vocabularies.

An ontology provides a rich expression of knowledge about a fragment. A body of formally represented knowledge necessary to index fragments is based on a conceptualization: an abstract, simplified view of the world that one wishes to represent. An ontology can be viewed as a closed indexing vocabulary, consisting of hierarchically organized concepts with attributes (e.g., synonyms, abbreviations), relations (is-a and part-of), constraints, and instantiations.

Available Indexing Vocabularies

Different communities of practice develop domain-specific indexing vocabularies. LOM allows the specification of domain-specific hierarchies of terms in attribute Taxon paths. For many domains however, there is no need to develop a domain ontology from scratch. In the context of the Semantic Web, highly semantic, ontology-based annotations play an important role (Berners-Lee, Handler, & Lassila, 2001). The Semantic Web, by using ontologies as indexing vocabularies, aims to add semantic terms to documents that cannot directly be derived from the content itself (Schreiber, Dubbeldam, Wielemaker, & Wielinga, 2001). The content in Web pages is given well-defined meaning and structure by means of indexes based on domain ontologies, and people are investigating how to define mapping schemas to integrate different domain ontologies.

Vocabularies that provide information about the material itself are often represented by an attribute for resource type. The Dublin Core Metadata Element Set (2003) recommends the following vocabulary for resource types: collection, dataset, event, image, interactive resource, service, software, sound, text, physical object, still image, and moving image. Texts are, for example, books, letters, dissertations, poems, newspapers, articles, and archives of mailing lists. In LOM, learning objects (including raw media data) are described in terms of learning resource type. Its values are: exercise, simulation, questionnaire, diagram, figure, graph, index, slide, table, narrative text, exam, experiment, problem statement, self-assessment, and lecture. Most available indexing vocabularies consist of flat, unordered lists of terms. However, structured hierarchies of terms may help indexers as well as retrievers to apply the values more precisely. For instance, values of LOM's learning resource type can be grouped into categories such as the presentation medium (slide), the way information is represented (graph, narrative text), or the roles objects play in instruction (exercise). Many other value lists, similar to that of the learning resource type in LOM, are proposed. For example, EMR provides example values for document format: hypertext, video, exercise, simulation, and questionnaire. National Geographic (2003) allows teachers to search on resource type, with values such as lesson plans, books/workbooks, videos, and software. Wiley and Cisco Systems, mentioned above, provide vocabularies to describe the semantic content and possible role of a fragment in instruction. Wiley provides a domain-independent taxonomy for inter-object comparison. This taxonomy allows a designer/developer to relate objects to each other, and avoids classifications such as file size in kilobytes that describe an object outside its context. Cisco System's approach is to annotate an information object as concept, fact, procedure, process, or principle (based on Merrill's Component Display Theory; Merrill, 1983).

The diversity of different value lists and the diversity of different dimensions addressed within value lists indicate that a standard annotation schema for fragments is desirable but has not yet been achieved. Standards are still under development and seem to lack the knowledge-rich and many-faceted aspects that are needed to optimize re-use through the Internet.

EMPIRICAL INVESTIGATION

Example of a Knowledge-Rich Annotation Structure

Based on experience gained in a European project IMAT (1) that aimed to re-use fragments of technical manuals in maintenance training design, an annotation structure that is shown in Table 1 was developed.

The concept behind this ontology is that the design task drives the re-use of material and content (Bystrom & Hansen, 2002). In design, basically, a learning goal, strategy, and activities are specified. A developer takes this blueprint and retrieves fragments that fit the design, using domain-related and semantic descriptions of the material. Although the IMAT indexing vocabulary was created for fragments of technical manuals that are re-used in maintenance training design for specific domains (e.g., the Renault Clio car), its attribute values are partly generic and are applicable across domains. Table 2 shows examples of generic and specific attribute values. In this example, an image showing, in steps, how to assemble the headlights of a car is annotated using structured value lists based on ontologies.

The IMAT indexing vocabulary aimed to optimize retrieval and support the choices developers make to judge the relevance of information fragments in the context of a design. This knowledge-rich indexing vocabulary was used in our empirical investigations. In the subsequent sections, we aim to answer the question of whether indexing vocabularies can be consistently applied to fragments by different people.

Consistency in Indexing

Having an indexing vocabulary does not guarantee that different people will index the same fragments in the same way. Consistency between indexers may depend on the nature of the indexing vocabulary (flat lists versus ontologies), properties of the domain (formal versus informal), media type of fragments (text, video, audio, pictures), and applicable attribute types (tangible versus intangible). This was investigated in two experiments.

Consistency of Indexing With Flat Value Lists: JAVA Experiment

In this experiment, subjects annotated a set of pre-selected fragments by assigning a single value to each attribute. To be able to judge subjects' annotations, these were compared with an expert's annotations; these were made by a professor of social science informatics with knowledge of the JAVA programming language, a great deal of pedagogical experience, and a long history of research in the area of ontologies. The subjects were inexperienced indexers. These were selected for two reasons: first, one cannot expect that in the future all indexing will be performed by experts; second, it is quite likely that users of metadata for retrieval will also be inexperienced. Therefore, a kind of worst case scenario was created, determining a bottom line for what can be achieved in terms of consistency.

The indexing vocabulary used in the JAVA experiment (see Table 3) comprised 11 LOM attributes and 4 additional ones (derived from the IMAT vocabulary). Value lists were flat, either scales (values ranging from low to high) or unordered collections of terms, except for the JAVA concept hierarchy, which was a structured value list.

In approximately 90 minutes, 13 groups of 28 subjects annotated the same 14 fragments. Subjects were third year social science informatics students; most had taken a JAVA course; most were familiar with search engines and had some instructional experience. Prior to the experiment, the subjects were trained in using the ontologies. Because using the JAVA indexing vocabulary is seen as a difficult task, the subjects were assigned to groups of two, and sometimes three, persons to pool knowledge of the domain. Subjects had as much time as was necessary to finish the indexing task. The fragments were texts and pictures, for example, a screen dump of a JAVA applet or a paragraph explaining a JAVA concept; the subjects were told these were suitable for an introductory JAVA course at a university level. To annotate the fragments, an indexing tool was developed that allowed subjects to browse through the fragments and to select a value from a drop down menu or to type in text when fixed lists were absent.

The dependent variable was the correspondence of subjects' annotations with those of the expert. Correspondence was made operational by calculating the percentage of subjects who annotated the same as the expert over all fragments. The results were analyzed based on strict matching criteria (i.e., an exact match between the subject's and the expert's value, free text annotations excluded), and on flexible matching criteria, to account for subjects who were less specific than the expert was but not necessarily wrong. The flexible matching criteria were that, in a structured value list such as the JAVA concept hierarchy, a subject's value matched the expert's more specific value. For instance, class matched with a direct sub-concept in the ontology applet. In value scales, two adjacent values matched (for example, very easy matched with easy), and in unordered value lists, similar values matched (for example, learning resource type value preface matched with foreword). The flexible matching criteria were a kind of approximation of an ontology, allowing a tentative answer to the question about the effect of structured versus flat value lists. If high consistency is found based on values that are similar but not exact, the same structured value lists, permitting different levels of abstraction, may be a means to reduce the number of mismatches between two indexers.

On average, less than half of the subjects' annotations strictly corresponded with the expert, and even with the flexible matching criteria, 22 percent of the subjects did not correspond with the expert's annotations. The flexible matching criteria resulted in an increase in consistency of 30 percent. A closer look at the results shows for which attributes a broad view (flexible matching criteria) is especially required. Table 4 shows the percentage of subjects that matched the expert's annotations. A high percentage indicates high correspondence. The rows show percentages based on the strict and flexible matching criteria. The columns represent attributes, ordered from high average percentages for the strict criteria (left column) to low.

Not surprisingly, attributes with tangible values that exclude each other (e.g., learning resource type and interactivity type) generated more agreement than abstract attributes such as semantic density because they required less interpretation. In this respect, the high percentages found for the abstract attribute knowledge type are surprising. This may be due to the fact that a computer language domain is highly structured and formalized, often containing clear definitions, which could make it easier to detect the type of knowledge in a fragment. Low average percentages were found for all attributes with value scales: difficulty, interactivity level, and semantic density. Interactivity level may have caused confusion because none of the objects was interactive. However, clearly judging on value scale turned out to be highly subjective. LOM anticipates on the subjective use of value scales, noting that: "inherently, this scale is meaningful within the context of a community of practice."

Despite the structured JAVA concept hierarchy (which could be improved), the correspondence found for domain-related annotations (topic) was below average, which could be an indication that the subjects' knowledge of JAVA was not sufficient. It could also be a consequence of the type of fragments that were annotated. Pieces of JAVA code and structural schemas were rather abstract, and sometimes complex, types of fragments and showed lower consistency (on average 40 percent) than a paragraph of text or a screen dump of an applet (on average 60 percent).

The JAVA experiment shows that with strict matching criteria, overall correspondence is low, indicating the difficulty of consistent indexing. Using a more flexible analysis, simulating an ontology, gives higher but not perfect consistency. Consistency is most difficult to achieve for attributes that require considerable interpretation to correctly assign a value to an object/fragment. In the next experiment, this last aspect was studied in more detail by providing structured value lists and allowing for multiple values. In addition, different media types were included.

Consistency of Indexing With Structured Value Lists: Gorilla Experiment

In this experiment, subjects annotated a set of pre-selected multi-media fragments in the domain of animals, using an indexing vocabulary with structured value lists (based on the IMAT indexing vocabulary). Subjects' multi-value annotations were compared with those of a team of experts, based on strict and flexible matching criteria, and the consistency between subjects' annotations that differed from that of the experts was also measured. Two experts, with the same qualifications as in the JAVA experiment, created the benchmark; they annotated the fragments independently and almost identically. The few discrepancies were discussed, leading to agreed upon values.

The indexing vocabulary that was used in this experiment was based on ontologies. It comprised five attributes with structured value lists, shown in Table 5.

This indexing vocabulary was designed to support the retrieval task of a developer and applies to small fragments. In approximately one hour, 21 subjects individually annotated the same 16 fragments (four texts, four pictures, four sounds, and four videos). Instead of a single value, at most five values per attribute were allowed. However, the number of terms was limited, to stimulate subjects to think about the most suitable terms. Subjects were told to be as precise as possible, yet they were allowed to use a super-concept where appropriate. The subjects (third year social science informatics students) represented inexperienced indexers, but were expected to have enough knowledge to perform the task individually (instead of in groups). A tool was developed for annotating the fragments. This tool displayed one of the fragments, for instance a short video of a chest-beating gorilla. Using previous and next buttons, subjects could switch between fragments. To index the fragments, concepts could be selected from an expandable ontology, and added to or removed from an annotation field. The 16 fragments were annotated per attribute, because concentrating on one attribute (one ontology) at a time requires less mental effort (Miller, 1956) than thinking about attributes alternately. The fragments were displayed in random order for each attribute to prevent a bias caused by the order in which fragments were presented.

The dependent variables were correspondence with experts' annotations and correspondence between subjects' annotations, excluding those that were the same as those of the experts. Analysis was based on strict and flexible matching criteria. The strict matching criteria represented a perfect match between two values. The flexible matching criteria permitted values of different levels of detail, where direct sub- or super-concepts of the experts' concepts were considered to match. For example, the values lowland gorilla and western lowland gorilla formed a matching pair using the flexible criteria, but not for the strict criteria. The scoring method calculated the base rate for an attribute, which is the prior probability that two annotations match. This scoring method corrected for coincidentally corresponding annotations. It is based on the probability of a match, given the number of concepts in the ontology, the number of concepts used by the experts and the number of concepts used by the subject. The following formula was used (hypergeometric distribution):

[FORMULA NOT REPRODUCIBLE IN ASCII]

Because a low probability of an accidental match should, when it occurs, be rewarded with a high score, the score a subject received for an attribute is 1-P (H=k). Each subject received a maximum score of 1 for each attribute that corresponded with the experts.

Contrary to the results of the JAVA experiment in which single value annotations were made with flat value lists, the difference between the analysis based upon the strict and the flexible matching criteria is small when multiple-value annotations are made with structured value lists. Based on the strict matching criteria, an average consistency of .71 was found, and based on the flexible matching criteria there was an average consistency of .73. This small difference shows that offering structured value lists leads to more consistency between indexers. Table 6 shows the average score across subjects, based on the flexible matching criteria. A high score indicates high correspondence with the experts' annotations. Rows represent sets of four objects of the same media type. Columns represent attributes, ordered from high (left column) to low average scores.

As in the JAVA experiment, high scores are found on tangible attributes. Representational type scores were significantly high (M=.89, t=10.74, df=22, p<.05). However, low scores were found on abstract attributes. Knowledge type scored significantly lower than other attributes (M=.53, t=-4.98, df=22, p<.05). Compared to the JAVA domain, the gorilla domain is far less formalized and structured, making it somewhat difficult to identify knowledge type unambiguously. Instructional role may have scored below average because subjects were relatively inexperienced instructional designers. For example, subjects used the concept illustration excessively to annotate images. This is an easy way out, because, in almost every case, images are included for illustrative purposes, independent of their precise content in terms of knowledge conveyed. Topic scores were almost similar to the ones in the JAVA domain using flexible criteria. Pictures scored lower than other media types. Instructional role and knowledge type of pictures scored significantly lower than average (M=.53, t=-6.12, df=21, p<.05; M=.39, t=-7.14, df=21, p<.05). In contrast, most subjects agreed with the expert on the topic of pictures.

To measure the degree of correspondence between subjects' annotations that were different from those of the experts, each possible subject pair was compared in both directions, using the same scoring method. Only the flexible matching criteria were applied: direct sub- and super-concepts were considered corresponding, except for root-concepts. Each subject received a maximum score of 1 for each attribute that corresponded with other subjects that were different from the experts. Table 7 shows the percentage of subjects that annotated with other values than the experts (%) and the average correspondence between those other values (M). Rows represent sets of four objects of the same media type. Columns represent attributes, ordered from low average percentage subjects that used alternatives (left) to high.

In the best situation, only a few subjects used different concepts than the experts (indicated by a low percentage); among the subjects who did, the same concepts were used (indicated by a high mean correspondence), as was the case for representational types of texts. Pictures, compared to other media types, were most often annotated differently from the experts (62 percent). Of all attributes, topics were most often annotated differently (87 percent); however subjects agreed on those other annotations to a large extent (.64). For the 55 percent of the subjects who used other knowledge type values than the experts, a correspondence of only .30 was found. These subjects disagreed with the experts and with each other. The rich topic structure of 86 concepts allowed for a high degree of agreement among subjects, whereas the flat list of only 10 knowledge type concepts may have hampered consistency among subjects' annotations. The lack of structure may have allowed subjects to build their own mental model of types of knowledge, leading to different viewpoints and interpretations of available concepts.

The results are combined in Table 8. Each cell shows three symbols. A '+' stands for the positive situation in which consistent indexing is feasible; a '-' indicates the negative situation. The first sign is positive (+) if the average correspondence between subject's and the experts' attribute values is .73 (the average from Table 6) or higher; the second is positive (+) if less than 50 percent of the subjects used alternative attribute values (from Table 7); the rightmost sign is positive (+) if the average correspondence between the subjects' different attribute values is .51 (the average) or higher (from Table 7).

Together, the comparison with experts and the comparison between subjects, provide insight into consistency in indexing material. Sounds and texts were relatively consistently indexed, video moderately, and pictures the least. A possible explanation is that pictures inherently require more interpretation than other media types. In contrast to other media types that take some time to read, view, or listen to, still images lack context and are likely to be interpreted in more than one way. Regarding attributes, description type was annotated fairly consistently. In a biological domain, subjects could easily discriminate between a physical and a behavioral description, as an example. Instructional role was moderately consistently indexed. To some extent, subjects had different mental models when it came to the roles fragments can play in instructional material, possibly due to a lack of instructional design expertise. 'Topic' was often annotated with multiple values, but the highly structured topic-concept hierarchy supported consistent annotation. 'Knowledge type' was least agreed upon. Subjects used alternative annotations on which they disagreed more than average. In this experiment, 'knowledge type' was the only attribute with a flat list of values. A structured value list is particularly desirable for this abstract attribute. 'Representational type' was, to a very high degree, consistently indexed. Typically, fragments of a small grain size are represented in a single excludable form; a fragment is either running text or a photo. To determine the 'representational type' one does not need much interpretation or abstraction, and a highly structured value list makes annotation even easier.

CONCLUSIONS

It was argued that from the perspective of re-use through the Internet, a learning object can best be defined as a relatively small piece of learning material about one (main) topic that consists of information fragments and instructional ingredients, and fits into a larger instructional structure such as a lesson. As standards for learning objects and theories for designing learning objects are under development, we are well on the way to supporting teachers, learners, and designers in their retrieval tasks. A stage is being reached where, similar to the Semantic Web approach, knowledge-rich, many-faceted annotations of elementary learning objects (fragments) are becoming important, as these can provide the necessary additional retrieval support for developers of learning objects. This can be achieved by using ontologies as indexing vocabularies. While it appears advantageous for developers and for people who must annotate material using a highly structured, rich indexing vocabulary, this is not per se an easier job than applying LOM's flat value lists. The aim of investigating consistency in indexing was to gain insight into the way people apply a standardized indexing vocabulary and how this is related to the nature of the domain, the types of attributes, and the media types involved.

Overall, different people annotate the same fragments with approximately 75 percent similarity. Given the relatively inexperienced subjects used, as a kind of worst case, this is a good result. The fact that consistency is considerably lower when annotations are made with unordered, flat lists of values indicates the positive effect of structure on consistency. In particular, when a single value is allowed and if strict matching criteria are used, consistency is low. Because of a lack of structure, values in unordered lists may have some overlap. Hence, more than one value may be applicable. Depending on the view of one indexer, a dominant value is selected, which may be a different value than that chosen by a next indexer. Value scales are used most inconsistently by different indexers. For example, values ranging from 'very easy' to 'very difficult' are applied in a subjective way and should, therefore, be avoided or broadly interpreted. Ontologies offer structured conceptual representations in 'is-a' and 'part-of' hierarchies and contribute to consistent annotations at a conceptual level. The effect of media type is that text, spoken or written, is more consistently indexed than pictorial material (images or video). Text is less poly-interpretable and, therefore, indexers have similar mental models and apply similar attribute values. The effect of type of attribute is that the degree of abstraction determines the extent to which different mental models are used. An abstraction is a simplification of reality, and people make different 'simplifications of reality' if there is no structure to support them. For abstract attributes that show very low consensus, either the ontology needs further specification or it should be accepted that some attributes are hard to annotate consistently. Finally, the effect of type of domain is less easy to establish. It seems that in more formal domains, such as programming languages, it is easier to be consistent about certain attribute types (particularly 'knowledge type') than in less formal domains.

Being aware of the degree of subjectivity employed in indexing and permitting multiple values would benefit retrieval. Flexible use of a structured indexing vocabulary leads to more consensus, but also to less specificity--a trade off about which one should be aware. From a retrieval perspective, specificity causes high precision, which might be more valuable to an author of instructional material than high recall.

Using ontologies brings the advantage of consistent indexes and high retrieval precision. In a broader sense, the effort and cost of indexing can be reduced by using efficient indexing methods and flexible tools. An example of an open and flexible indexing and query tool is one developed for ARI-ADNE (Neven, Duval, Ternier, Cardinaels, & Vandepitte, 2003) in which the indexer can specify a profile that allows the setting of default values that are not likely to change. Efficient use of metadata is necessary because indexing is labor intensive and expensive, in particular, when many small objects have to be annotated instead of a few large ones. We are currently investigating the cost effectiveness of extensive indexing and re-use of fragments in instructional material. Automatic semantic indexing is the ultimate goal, thus reducing both high cost and inconsistency. Until that is achieved, problems that come with manual indexing, such as inconsistencies, remain and must be made known to the people involved.

Table 1 IMAT ontology

Category     Attribute

Domain       Topic
Material     Media Type and Format
             Representational Type
             Structural Type
             Description Type
             Description Scope
             Knowledge Type
             Instructional Role
Design task  Learning Goal
             Instructional Strategy
             Instructional Activity
             Instructor Action
             Learner Action

Table 2 Examples of generic and specific values in the IMAT indexing
vocabulary

             Generic example values           Specific example values

Domain       Applies to systems in general    Specific for cars
             Topic: - System,                 Topic: - Car,
                      - Subsystem,                     - Body,
                        - Unit                           - Headlight
                                                           carrier panel
Material     Applies to digital fragments     Specific for technical
             Media type: - Image,               manuals
                           - .jpg             Description Type:
             Structural Type: - Figure          - Depiction of a
             Representational Type:               component
               - Pictorial representation         - Cross section of a
                 - Structured representation        component
                   - Chart
             Description Type: - Structural
                                 overview
             Description Scope: - Single
                                  elaborate
Design task  Applies to instructional design  Specific for maintenance
             Knowledge Type: - Procedural       training design
                               knowledge      Learning Goal:
             Instructional Strategy:          - Being able to maintain
               - Modeling                       system X
             Instructional Activity:            - Skill in performing
               - Giving examples                  inspection
             Instructor Action: - Provide         - Skill in taking
                                  example,          precautions
             Learner Action: - Comprehend     Knowledge Type:
                               example        - Procedural knowledge
             Instructional Role of the        - Knowledge in taking
               fragment: - Learning block         precautions
                          - Illustration
                           - Example

Table 3 Indexing vocabulary used in the JAVA experiment

Category   Attribute                           Example values

General    LOM: Title                          Free text
           LOM: Description                    Free text
Lifecycle  LOM: Version                        Free text
           LOM: Contribute                     Free text
Content    LOM: Learning Resource Type         Narrative text
           (adapted to the JAVA domain)        Programming code
           LOM: Semantic Density               Very low
                                               Very high
           Description Type (writing           Physical description
           perspective)                        Procedural description
           Description Scope (nature of grain  General (introductory)
           size)                               Detailed
           Knowledge Type (kind of knowledge)  Terminological knowledge
                                               Conceptual knowledge
                                               Practical knowledge
Education  LOM: Learning Context               School
                                               Higher education
           LOM: Interactivity Type             Active
                                               Expositive
           LOM: Interactivity Level            Very low
                                               Very high
           LOM: Difficulty                     Very easy
                                               Very difficult
           Instructional Role (possible        Example
           role(s) in instruction)             Explanation
                                               Good practice
Domain     LOM: Taxon Path (JAVA domain        Class
           ontology comprising an is-a and a   Method
           part-of concept hierarchy)          Variable

Table 4 Percentage subjects corresponding with the expert's annotations

          Learning Resource  Interactivity  Description  Knowledge
          Type               Type           Type         Type

Strict    84%                69%            56%          55%
Flexible  95%                86%            81%          80%

          Description  Topic    Instructional  Difficulty  Topic is-a
          Scope        part-of  Role

Strict    49%          46%      45%            45%         44%
Flexible  75%          75%      74%            63%         77%

          Learning  Semantic Density  Interactivity Level  Average
          Context

Strict    37%       26%               16%                  48%
Flexible  87%       56%               89%                  78%

Table 5 Indexing vocabulary used in the gorilla experiment

Attribute                             Example values

Representational Type (resembles      Pictorial representation
LOM's learning resource type)             Reality-related representation
                                          Photo
                                      Textual representation
                                          Structured text
                                          List
Topic (gorilla concept hierarchy)     Animal
                                          Gorilla
                                          Mountain Gorilla
                                      Behavioral property
                                          Social behavior
                                          Communication
Description Type                      Organizational description
                                      Content description
                                          Behavioral description
                                          Physical description
Knowledge Type                        Conceptual knowledge
                                      Definitional knowledge
                                      Factual knowledge
                                      Practical knowledge
Instructional Role                    Pre-instructional
                                          Advance organizer
                                          Motivation
                                          Introduction
                                          Learning Goal
                                          Definition
                                      Learning block
                                          Illustration
                                          Example

Table 6 Correspondence between subjects' and expert's annotations

          Representa-  Topic  Description  Instructional  Knowledge
          tional Type         Type         Role           Type

Average       .89       .77      .76          .70            .53
Pictures      .84       .89      .71          .53            .39
Sounds        .89       .76      .74          .82            .56
Texts         .95       .70      .81          .69            .61
Videos        .87       .74      .77          .76            .55

          Average

Average     .73
Pictures    .67
Sounds      .75
Texts       .75
Videos      .74

Table 7 Correspondence between subjects' alternative annotations

          Representational  Description  Knowledge  Instructional
              Type             Type        Type         Role
          %            M    %       M    %     M    %      M

Average   25%          .68  42%     .45  55%   .30  56%    .48
Pictures  41%          .57  45%     .37  70%   .37  68%    .57
Sounds    15%          .56  42%     .51  50%   .17  40%    .36
Texts     13%          .96  35%     .38  50%   .30  67%    .52
Videos    30%          .60  44%     .52  50%   .37  49%    .47

           Topic    Average
          %     M   %     M

Average   87%  .64  53%  .51
Pictures  86%  .72  62%  .52
Sounds    88%  .58  47%  .44
Texts     87%  .66  51%  .53
Videos    86%  .60  52%  .51

Table 8 Practicability of consistent indexing

         Representa-  Description         Instructional
         tional Type  Type         Topic  Role

Picture  + + +        - + -        + - +  - - +
Sound    + + +        + + +        + - +  + + -
Text     + + +        + + -        - - +  - - +
Video    + + +        + + +        + - +  + + -
Average  + + +        + + -        + - +  - - -

         Knowledge
         Type       Average

Picture  - - -      - - +
Sound    - - -      + + -
Text     - - -      + - +
Video    - - -      + - +
Average  - - -

Acknowledgements

We would like to take this opportunity to thank Daniel Rehak for contributing to improving this article.

Note

(1) Integrating Manuals and Training (de Hoog et al., 2002a and 2002b).

References

Anjewierden, A. & Kabel, S. (2001). Automatic indexing of documents with ontologies. In: Proceedings of the 13th Belgium-Netherlands Conference on Artificial Intelligence (BNAIC) 2001, Amsterdam, 23-30.

ARIADNE Educational Metadata Recommendation Version 3 (2002). [Online]. Retrieved July 20, 2003 from: http://www.ariadne-eu.org.

Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The Semantic Web. Scientific American, 284(5):34-43.

Bystrom, K., & Hansen, P. (2002). Work tasks as unit for analysis in information seeking and retrieval studies. In: Proceedings of the 4th International Conference on Conceptions of Library and Information Science: Emerging Frameworks and Methods (CoLIS4), Seattle, Washington, 239-252. Greenwood Village, CO: Libraries Unlimited.

Cisco Systems (2001). Reusable Learning Object Strategy, version 4.0. [Online]. Retrieved July 20, 2003, from: http://business. cisco.com.

De Hoog, R., Wielinga, B. J., Kabel, S., Anjewierden, A., Verster, F., Barnard, Y., DeLuca, P., Desmoulins, C. & Riemersma, J. (2002a). Re-using technical manuals for instruction: document analysis in the IMAT project. In: Intelligent Tutoring Systems (ITS) 2002, Workshop proceedings Integrating technical and training documentation, San Sebastian, Spain, 15-27.

De Hoog, R., Kabel, S., Barnard, Y., Boy, G., DeLuca, P., Desmoulins, C., Riemersma, J. & Verstegen, D. (2002b). Re-using technical manuals for instruction: creating instructional material with the tools of the IMAT project. In: Intelligent Tutoring Systems (ITS) 2002, Workshop proceedings Integrating technical and training documentation, San Sebastian, Spain, 28-39.

Dublin Core Metadata Element Set Version 1.1. (2003). [Online]. Retrieved January 20, 2004, from: http://dublincore.org/documents/dces/.

IEEE Standard for Learning Object Metadata, IEEE-SA Standard 1484.12.1-2002, IEEE Standards Department, Institute of Electrical and Electronics Engineers, Inc.

Kabel, S., de Hoog, R., Wielinga, B.J., & Anjewierden, A. (2004). The added value of task- and ontology-based markup for Information Retrieval. Journal of the American Society for Information Science and Technology, 55(4), 348-362.

Merrill, M. D. (1983). Component Display Theory. In: Reigeluth, C. M., ed., Instructional Design Theories and Models: An overview of their current status (Vol I), Hillsdale, NJ: Lawrence Erlbaum Associates.

Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97.

Mohan, P. & Greer, J. (2003). Reusable learning objects: current status and future directions. In: Proceedings of the ED-Media World Conference on Educational Multimedia, Hypermedia and Telecommunications, June 24, 2003; Honolulu, Hawaii, USA, 257-263.

National Geographic (2003). National Geographic Web site, last accessed July 20, 2003: www.nationalgeographic.com/education/.

Neven, F., Duval, E., Ternier, S. Cardinaels, K., & Vandepitte, P. (2003). An open and flexible indexation- and query tool for Ariadne. In: Proceedings of the ED-Media World Conference on Educational Multimedia, Hypermedia and Telecommunications, June 24, 2003; Honolulu, Hawaii, USA, 107-114.

Polsani, P.R. (2003). Use and Abuse of Reusable Learning Objects. Journal of Digital Information, Vol. 3, Issue 4, Article No. 164. [Online]. Retrieved, January 20, 2004, from: http://jodi.ecs.soton.ac.uk/Articles/v03/i04/Polsani/.

Schreiber, A.Th., Dubbeldam, B., Wielemaker, J., & Wielinga, B. J. (2001). Ontology-based photo annotation. IEEE Intelligent Systems, May-June 2001, 66-74.

Wiley, D. A. (2000). Learning Object Design and Sequencing Theory. Doctoral dissertation. [Online]. Retrieved July 20, 2002 from: http://wiley.ed.usu.edu/docs/dissertation.pdf.

SUZANNE KABEL, ROBERT DE HOOG BOB WIELINGA AND ANJO ANJEWIERDEN

University of Amsterdam

Human Computer Studies Laboratory

The Netherlands

[e-mail]

[e-mail]

[e-mail]

[e-mail]