Reusing educational material for teaching and learning: current approaches and directions.

In this article we survey some current approaches in the area of technologies for electronic documents for finding, reusing, and adapting documents for teaching or learning purposes. We describe how research in structured documents, document representation and retrieval, semantic representation of document content and relationships, and ontologies could be used to provide solutions to the problem of reusing educational material for teaching and learning.

**********

E-learning involves different aspects of using electronic documents for learning-related activities. It ranges from managing curriculum courses on the Web (advertising, registration, scheduling, exams, etc.), to online classes, publishing course material for the students, and dedicated online tutorial systems. A lot of effort has been dedicated to creating high-quality and relevant online learning material, as well as the design and implementation of systems that support users in their learning process. Recent research has focused on adaptive learning environment that can personalise the learning experience.

However, as pointed out by Casey and McAlpine (2003), "anyone who has had to create learning materials from scratch knows just how labour intensive and time consuming the process can be, even with the existence of a detailed course descriptions and lesson plans. This creative process can be made easier by the reuse of existing teaching and learning materials."

Preparing learning material typically involves:

* finding good document sources relevant to the topics and to the audience;

* selecting more specific parts of documents that could be reused, in particular graphics, tables, images, which have a high illustrative power, and creating new material that can be adapted for personalisation and future reuse;

* defining the sequence in which documents and fragments about some concepts should be accessed or presented (prerequests); and

* defining the curriculum planning that would fit with the pedagogic approaches, and that will hopefully adapt to the actual learner.

In this article we survey how technologies for electronic documents are being used for finding, creating, and adapting material for teaching and learning purposes. We try to identify current approaches and future directions that could support the reuse of existing curriculum material as well as instructional design.

The article is organised as follows: first a section on indexing and finding existing relevant educational material; the next section is concerned about the creation, retrieval, adaptation, and assemblage of fragments of documents; followed by a section on contrasting the navigation and access capabilities offered in tutoring systems compared to open learning environments; next we study how to integrate textual material with active components such as programs; then we offer some directions to define and implement reusable instructional design, and finally we present our conclusions.

FINDING EXISTING DOCUMENTS

Nowadays many documents can be found on the Web and used for self-learning. For example there are online tutorials, basic and advanced courses, opinions and advice, book references, and research papers.

Search engines such as Google rank documents that are pointed to by other web pages (implicit recommendation). A typical example would be asking Google for Java tutorials and getting back what look like very good answers on the first page only: you can choose between the Sun tutorials, the IBM pages, the Java Cafe, and so forth, or you may prefer to start with the hub assembled by Marty Hall (from the Johns Hopkins University Applied Physics Lab), or the online tutorial by Richard G. Baldwin.

However, it is very difficult to select the best document or references amongst so many answers and some extra time must be devoted to assess the quality of the documents, for example, by looking at the qualifications of the authors and cross-references using CiteSeer, or reading recommendations by other users. You may also have to carefully check for copyright statements or licence agreements before using documents and software. Furthermore, some sites that offer "distance learning courses" are effectively scams that pretend to offer real academic courses and diplomas. However these diplomas are false and often the material is scant and ill prepared.

To provide high quality learning material, many educational bodies have created Educational Libraries that index the learning material using metadata that can support a more precise selection. Examples of such libraries are the Gateway to Educational Materials (GEM) (1) and WebCT (2) in the US, Careo (3) in Canada, EdNA (4) and LRC (5) in Australia, ARIADNE (6) and School-NET (7) in Europe.

The advantage of these digital repositories over the Web is that, like classical libraries, they hold much more metadata on each of the resources that can help students, teachers, and systems to retrieve more relevant documents than with full text search. Some of them, such as Merlot also include annotations and peer reviews.

However, there is no universal metadata standard for learning materials and many different standards such as IMS (9) [2], UKOLIN (10), and LOM (11) are being used. For comparison between metadata standards for education see Easel (2002). The Dublin Core metadata is a first attempt to build a simple common standard for resource discovery on the Web. The Dublin Core Educational Working Group (DCMI) (12) has recognised the need for adding to the 15 core elements some elements specific to educational purposes, such as "Audience" (who would benefit the material), "conformsTo" (learning objectives), "Pedagogy" (process to achieve the learning objective), and "Quality." "Quality," sometimes replaced by "Standard," is aimed at certifying that the material has been evaluated for educational purpose by some recognised body.

The Resource Description Format (RDF) could provide a higher level description where documents and concepts can be linked together, as well as concepts between themselves. Amann, Fundulaki, and Scholl (2000) have proposed to query a digital library through an ontology and a thesaurus that have been integrated using an RDF format. This provides a rich description to the resources that can be shared by the community. Carmichael (2002) advocated the importance of the "assessment for learning" in describing reusable educational resources. He is using the Dublin Core qualified for that purpose but also RDF metadata to describe classroom activities and their relationships to broader educational strategies. We will come back to educational strategies and instructional design in a later section.

ASSEMBLING FRAGMENTS OF DOCUMENTS

In recent years, a lot of research has been dedicated to developing flexible learning material that can deliver personalised courses depending of a number of factors such as the user's learning preferences, his current knowledge based on previous assessments, or previous browsing in the material. Authoring such courses requires the authors to define reusable chunks of documents that can be retrieved, adapted, and assembled in a coherent way for a given educational purpose.

De Bra and Calvi (1998) created an adaptive hypermedia system (AHA), where the content of pages is adapted to the user by assembling fragments and fragment variants. The user model is created dynamically based on which pages the user has already read and which problems have been successfully solved.

More sophisticated approaches for dynamically generating or assembling coherent pages involve natural language generation are in Peba II (Milosavljevic, 1997) or Tiddler (Wilkinson, Lu, Paradis, Paris, Wan, & Wu, 2000). In Peba II, comparisons between animals are generated on the fly depending on the user and the animal descriptions that have already been read. In Tiddler the selection of the fragments and the coherence of their composition, including natural language text generation, is driven by a task-driven discourse model. If the task was a learning task, the discourse model could reflect the instructional steps defined by the chosen instructional design.

Virtual documents are based on declarative specifications for retrieving and dynamically assembling fragments of existing documents (Vercoustre & Paradis, 1997). Personalised virtual documents used in educational systems select fragments based on the user model and rich semantic descriptions of the fragments (Iksal, 2002). A common approach in the personalised virtual document community (13) is to describe fragments in term of concepts that are part of a domain or application ontology. Concepts are related to each other by standard ontology relationships as well as prerequisites. A concept cannot be learned before prerequested concepts are all understood. Consequently document fragments related to a concept will not be proposed by the system before fragments related to prerequested concepts have been accessed. In more intelligent learning systems, tests are proposed to the user to check whether the concepts are sufficiently understood. We will come back to this aspect in a later section.

Unfortunately, the way fragments are described and used is very much system and application dependant and cannot be reused by another system for another learning experience on the same topic but with a different objective, or a different instructional method. Most often the fragments have to be written from scratch with the particular application in mind.

Learning Object

An attempt to overcome this problem is to define and create learning objects. This is the objective of the IEEE's Learning Object Metadata (LOM) project (14) who gives this definition of learning objects: "A learning object is any resource or content object that is supplied to a learner by a provider with the intention of meeting the learner's learning objective(s) .... and is used by the learner to meet that learning objective(s)."

An important aspect of the LOM model compared to library catalogues is that it incorporates metadata relevant to curriculum design and teaching methodology in addition to descriptions of content and authorship. It uses standards such as DC or IMS and extends them to describe learning objects in a similar way to the Dublin Core Educational Working Group (DCMI) for full documents, but with a stronger focus on the learning objectives.

The LOM project also recognises that "learning content has generally been developed in conjunction with some sort of learning system that keeps track of learners. As the learners interact with the content results are passed back to the system. If the system allows it, the content can also change its behaviour based on learner information stored in the system."

Although intended to be reusable, the learning objects do not carry with them the instructional structure in which they should or could be used. The instructional design is traditionally contained in the document itself. This is lost when the document is broken into small objects and must be hard coded in each learning system that reuses them. Jonassen and Churchill (2004) questioned whether there is any learning orientation in learning objects. Of course this would depend on the granularity of the learning objects. If they are large objects (documents) that contain their self argumentation then there is a need for accessing the internal learning objects, possibly with Open Learning Objects (OLOs) as proposed by Shi, Rodriguez, Chen, and Shang (2004).

Thus, it would be important to be able to reuse parts of documents that have been written as self-contained learning documents and carry with them their full argumentation model.

XML Retrieval

An alternative to independent learning objects described by external metadata is to create teaching and learning materials that contain enough information that allows them to be reused in new situations. To achieve this we need the materials to be structured in such a way that we can also retrieve their smaller constituent parts (i.e., parts of individual lessons).

Describing learning objects and documents in XML could help making them more reusable and adaptable. First, XML can make the structure of reusable chunks explicit and automatically processed. Second, it preserves the context in which a fragment has been created and can be made available to teachers and students to help understanding the value of the fragments.

Examples can be drawn from the experience with the INEX working group on XML search evaluation (Fuhr, Malik, & Lalmas, 2003). In its first year, INEX working groups took on a series of retrieval tasks (queries) on a large collection of XML documents (about 12,000 articles in the IEEE Computer Society publications since 1995). One of the proposed topics involved "finding figures about the Corba architecture and the paragraphs that refer to them."

It is well recognised that document elements such as figures or tables can be more concise than a long discourse and have high pedagogical value. However, a figure without its caption is hardly understandable and often requires complementary information. Good XML retrieval engines should be able to retrieve such elements (and rank them) while providing some context, or the full embedding document as part of the answer.

This example was taken from an XML collection for which the DTD is very much publishing oriented and does not contain many tags that are semantically significant (such as figures, tables, bibliographic references). Its tags are mostly structural, such as section, paragraphs, lists, and so forth, and, in this case, more explicit metadata may be required for fragments of documents to be directly used in a learning environment.

Another drawback in querying XML documents is the possible heterogeneity of the DTDs for different collections. It should not be expected that the users, or even a given learning system, could know the actual tags used in different collections. Amann, Fundulaki, Scholl, Beeri, and Vercoustre (2001), have proposed to query XML collection through an ontology where concepts and relations in the ontology have been mapped to fragments of XML documents.

More semantic metadata can also be attached to fragments of existing XML documents (when preparing a new course) using RDF description and URIs that refer to those fragments (e.g., using Xpaths). The RDF metadata are then seen as external annotations to the material and different authors can create their own, or possibly reuse existing ones. This is the approach taken in ELM-ART (Brusilovsky, Schwarz, & Weber, 1996) where flexible and personalised browsing is built upon existing documents.

TUTORING SYSTEM VERSUS OPEN LEARNING ENVIRONMENT

In traditional books and textual documents, the organisation of the learning material is decided by the author and the learner is expected to read the document linearly, although nothing prevents him to jump to the conclusions first or to skip a section if he is already familiar with the concepts. The flexible nature of hypertexts and online materials offers new opportunities and challenges for learning support that can guide the learner in a more personalised way. In particular, when the content is split into smaller units, the learning system is expected to provide some guidance as to which part to read next.

Eklund, Brusilovsky, and Schwarz (1997) have developed "Interbook" that provides adaptive navigation support. The system records previous user's navigation to infer what knowledge the user has already acquired and suggest links to access other pages based on the prerequisites for those pages. Eklund, Brusilovsky, and Schwarz (1998) studied the use of link annotation in educational hypermedia, while De Bra and Calvi (1998) discussed the use of colour link annotations and link hiding to provide better guidance to the learner (De Bra & Calvi, 1998). They compared learning interfaces where only a "next" button is provided with interfaces where a broader choice is offered. They concluded that, in this particular experiment, beginners may prefer a strong guidance while more experienced learners would access more material with more open choices.

Intelligent Tutorial Systems, as their name suggests, are designed to provide strong support to the learner and try to propose to the user only the best recommendation for the next step in the learning process. However, Hubscher, and Puntambekar (2001) questioned the positive learning effect of very strict guidance, arguing that "more guidance does not necessarily result in more learning." Instead of embedding the macro-structure in the text with hyperlinks, they proposed that the reader's learning process can be more successfully supported with meta-level tools such as concept maps. A concept map presents ideas in the form of nodes which are linked by a word representing a concept. Concept maps are very powerful in helping students see the numerous relationships between concepts and enforce the learning process at a higher level.

Bunt, Conati, Hugget, and Muldner (2001) suggested that Open Learning Environments can be more beneficial for learning than tutor-controlled systems because of the active role the learner plays in knowledge acquisition. They proposed to place less emphasis on explicit instruction and more on providing the learner with tools that support learning through unconstrained exploration of the target instructional domain. However, their system also monitors the users and tries to detect when they experience difficulty. The system provides more guidance only when necessary.

In Open Learning Environments, it is possible to reuse and integrate more material that have been created in other contexts since the system does not have to make strict choices on what to read next; alternatives can be offered. However, it is still very important that a good description of the underlying material is available to the system to automatically generate good concept maps or other meta-level browsing support.

What is missing at this level is a standard way of describing concept maps and, more generally, how the information is related according to instructional intention and strategies.

Problem Solving and Active Examples

So far we have only mentioned textual material (documents, fragments of documents, and their hyper textual organisation) for composing the learning material, that is, material that the learner would read and be expected to understand before going further.

However most online tutor systems also include tools to verify that the user has effectively learned what it was supposed to learn. The user can be asked to answer a few questions, to solve a problem or to write a program (De Bra & Calvi, 1998; Brusilovsky et al., 1996). This allows the system to dynamically update the user model more accurately than just based on documents that the user has previously accessed.

Learning environments therefore have to intermix documents with more active components. Although not many standards have been developed for supporting it, the idea has been presented before as literate programming. In 1992, Donald E. Knuth introduced literate programming, a methodology that is defined as the combination of documentation and program source together in a fashion suited for reading by human beings. He created the original literate programming tool called WEB, which he used to write TeX and MetaFont.

The idea is that the documentation used for learning a programming language should include active examples of what the language offers. By active we mean examples that the user can test and get results that are immediately included in the embedding document. In this vision, "a program is also a document that teaches programming to the reader through its own example."

A recent XML-based proposal could become a standard way to include programs and activable components into teaching material. Active XML is currently developed for supporting the activation of services from XML documents and returning their value under the form of XML data that can be included into the initial document (Abiteboul, Benjelloum, Manolescu, Milo, & Weber, 2002). Although Active XML is very new and not standard, a similar approach could lead to more reusable and rich learning material.

Instructional Design

So far we have discussed how information can be reused based on its content. As described in previous sections, existing approaches annotate fragments or learning objects with semantic descriptions taken from an ontology of concepts. A concept cannot be learned before prerequested concepts are all understood. If we assume for the moment that standard ontologies are accepted for specific domains then we can imagine a system and/or author that is able to coherently reuse fragments created by others.

However, such a system or author is limited to reusing the fragment within the implicit instructional intent of the original author. If, for example, we create fragments consisting of (a) a diagram illustrating the parts of an engine and (b) a photograph of an engine and describe both with concept-based content metadata such as "engine," then these fragments can only be retrieved (for reuse) with a general query. Human inspection will be required to decide on the most appropriate fragment for reuse in the new course.

Kabel, de Hoog, Wielinga, and Anjewierden (2003; Delestre, Pecuchet, and Greboval (1999), noted that to make information truly reusable for teaching then information fragments need to be annotated with descriptional and instructional metadata as well as content or domain metadata. If we annotate the fragments further as (a) engine: schematic representation (specific): theoretical knowledge (illustration), and (b) engine: photo (specific): factual knowledge (example) then we can make specific instructional and domain queries when constructing the course.

Going in that direction, Tutor (Czarkowski & Kay, 2001) uses an Adaptive Teaching Mark-up language to describe the course maps, the learner parameters and a set of lessons (the actual teaching material) that may be adapted.

The University of Passau in Germany has developed a didactical reference model, a teachware model and a mark-up language based on Instructional design (Suss, Freitag & Brossler, 1999). The teachware model describes the modular structure of the learning content, while the didactical model describes its didactical structure that can reflect different pedagogical model using the same material.

To use such marked-up data a rich set of instructional strategies are required along with the conditions in which they are appropriate. Curriculum authoring should be supported by good instructional designs established by Instructional Science.

Instructional Science is based on the psychology and sociology of learning and consists of theories, models, and methodologies for instruction and contains both descriptive and prescriptive components--the latter forms part of what is called instructional design. Instructional design is domain independent and theory based. The use of such knowledge will be required in writing instruction-aware learning systems and it may be that RDF (in the form of DAML+OIL--may be used to represent this knowledge in both a Human readable and machine readable form.

For these strategies to be related to the instructional intention of the authored information fragments, both the fragments and the instructional strategies need to be "ontology-aware" (Mizoguchi & Bourdeau, 2000). An instructional ontology includes concepts such as the learning goal, definitions, background, example, explanation, reminder, and so forth.

Describing instructional strategies with RDF-based ontologies will allow both authors to manually implement these strategies or adaptive systems to automatically process them. In this area the Ontology Inference Layer OIL is a proposal for a web-based representation and inference layer for ontologies (Fensel, Horrocks, Van Harmelen, Decker, Erdmann, & Klein, 2000). It combines the widely used modelling primitives from frame-based languages with the formal semantics and reasoning services provided by description logics. It is compatible with RDF Schema (RDFS) and includes precise semantics for describing term meanings (and thus also for describing implied information).

The DARPA Agent Markup Language (DAML) is an effort to develop a language and tools to facilitate the concept of the semantic web. The DAML group pooled efforts with the Ontology Inference Layer to propose DAML+OIL, a language for expressing far more sophisticated classifications and properties of resources than RDFS. DAML+OIL is a current W3C proposal (www.w3.org/Submission/2001/12/) for a semantic markup language for web resources. Some current research is looking at building reasoning support for the language (Broekstra, Klein, Decker, Fensel, & Horrocks, 2000).

Conclusion

We have surveyed research in the area of technologies for electronic documents and shown that there are many relevant areas that the AIED community could draw on to allow educational material to be reused when creating a new course, whether that is done by an author or a system. In particular:

* Electronic document technologies can provide standard formats for describing curriculum material and associated metadata at different levels of granularity.

* While XML can provide a rich format for describing fully authored documents that support extraction of fragments, RDF provides a flexible and rich description for selecting and combining fragments to provide a more personalised learning experience.

* Active XML may provide a standard way to augment standard passive course material by embedding and activating problem solving modules into the learning material.

* To take advantage of instructional design, based on the psychology and sociology of learning, we need to represent instructional strategies in both human and machine readable form. The problem of representing instructional intention for educational material and being able to use it through appropriate application of instructional strategies may be resolved by drawing on ontology research; work in the semantic web with the DAML+OIL W3C submission appears to be particularly relevant. As tools appear that can reason with fragments of information marked up with DAML+OIL we may see the emergence of authoring environments that help the teacher compose new courses based on existing material and her teaching style. Eventually we would hope to see automated learning environments that are able to construct new curricula based on a learner's domain request and instructional preference through the reuse of existing educational material.

Notes

(1) GEM (The Gateway to Educational Materials), http://gem.syr.edu/

(2) WebCT, www.webct.com/otl

(3) Careo (Campus Alberta Repository of Educational Objects), Canada, www.careo.org

(4) EdNA (Education Network Australia), http://www.edna.edu.au/metadata/

(5) LRC (Learning Resources Catalogue), http://www.hkulrc.unsw.edu.au/

(6) ARIADNE, Alliance of Remote Instructional Authoring and Distribution Networks for Europe, http://www.ecotec.com/sharedtetriss/projects/files/ariadne.html

(7) SchoolNet(Europe), http://www.eun.org/portal/index-en.cfm

(8) MERLOT (California), http://www.merlot.org/Home.po

(9) IMS (Instructional Management System Global Learning Consortium) standards for teaching and learning materials. http://www.imsproject.org/

(10) UKOLN Metadata for Education Group, http://www.ukoln.ac.uk/metadata/education/

(11) IEEE Learning Technology Standards Committee (LTSC), http://ltsc.ieee.org/wg12/

(12) DCMI Education Working Group, http://dublincore.org/groups/education/

(13) Workshop on Documents Virtuels Personnalisables, 2002, http://iasc.enst-bretagne.fr/DVP2002/programme.htm

(14) Learnet, http://learnet.hku.hk/objects.htm

References

Abiteboul, S., Benjelloum, O., Manolescu, I., Milo, T., & Weber, R (2002, August). Active XML: Peer-to-peer data and web services integration. Proceedings of the 28th International Conference on very Large Data Bases (VLDB 2002), (pp. 1087-1090), Hong Kong, China.

Amann, B., Fundulaki, I., & Scholl, M. (2000). Integrating ontologies and thesauri for RDF schema creation and metadata querying. International Journal of Digital Libraries (JODL), 3(3), 221-236

Amann, B., Fundulaki, I., Scholl, M., Beeri, C., & Vercoustre, A.-M. (2001, May). Mapping XML fragments to community web ontologies. Proceedings of the Fourth International Workshop on the Web and Databases (WebDB'2001). Santa Barbara, CA.

Broekstra, J., Klein, M., Decker, S., Fensel, D., & Horrocks, I. (2000, September). Adding formal semantics to the web: Building on top of RDF schema. Proceedings of the ECDL Workshop on the Semantic Web (SemWeb2000), Lisbon, Portugal. Retrieved from http://www.ics.forth.gr/isl/SemWeb/proceedings/session2-2/paper.pdf

Brusilovsky, P., Schwarz, E., & Weber, G. (1996). ELM-ART: An intelligent tutoring system on World Wide Web. Proceedings of the Third International Conference on Intelligent Tutoring Systems, ITS-96, Berlin, Retrieved from http://www.psychologie.uni-trier.de:8000/projects/ELM/elmart.html

Bunt, A., Conati, C., Hugget, M., & Muldner, K. (2001). On improving the effectiveness of open learning environments through tailored support for exploration. Proceedings of AIED 2001, 10th World Conference of Artificial Intelligence and Education, San Antonio, Texas. Retrieved from http://www.cs.ubc.ca/%7Econati/my-papers/aied2001.pdf

Carmichael, P. (2002) Learning how to learn: Using the Dublin core metadata element set to support teachers as researchers. Proceedings of the International Conference on Dublin Core and Metadata for e-Communities, (pp. 201-203).

Casey, J., & McAlpine, M. (2003). Writing and using reusable educational materials--a beginner's guide. Retrieved from http://www.gla.ac.uk/rcc/staff/mhairi/index.html

Czarkowski, M., & Kay, J. (2001). Tutor: Support for scrutably personalised documents. Proceedings of the 6th Australian Document Computing Symposium (ADCS 2001). Retrieved from http://www.ted.cmis.csiro.au/adcs01/011_Czarkowski.pdf

De Bra, P., & Calvi, L. (1998). AHA! An open adaptive hypermedia architecture. The New Review of Hypermedia and Multimedia, 4, 115-139.

Delestre, N., Pecuchet, J-P., & Greboval C. (1999) How to design an adaptive dynamic hypermedia for teaching? Proceedings of AI-ED'99, (pp. 654-656) Le Mans, France.

Educator Access to Services in the Electronic Landscape (EASEL) (2002). Evaluation Report, EASEL--IST Project 10051 Report, March 2002.

Eklund, J., Brusilovsky, P., & Schwarz, E. (1997, July). Adaptive textbooks on the WWW. Proceedings of the Third Australian Conference on the World Wide Web (AUSWEB97), Queensland, Australia.

Eklund, J., Brusilovsky, P., & Schwarz, E. (1998). A study of adaptive link annotation in educational hypermedia. Journal of Universal Computer Science, 4(4), 429-448.

Fensel, D., Horrocks, I., Van Harmelen, F., Decker, S., Erdmann, M., & Klein, M. (2000). OIL in a nutshell. Lecture Notes in Artificial Intelligence. In R. Dieng (Ed.), Proceedings of the 12th European Workshop on Knowledge Acquisition, Modeling, and Management (EKAW'00), (pp. 1-16). Berlin/Heidelberg, Germany: Springer-Verlag.

Fuhr, N., Malik, S., & Lalmas, M. (2003, December). Overview of the initiative for the evaluation of XML retrieval (INEX) 2003. In N. Fuhr, M. Lalmas & S. Malik (Eds.) Proceedings of the 2nd Workshop of the Initiative for the Evaluation of XML Retrieval (INEX), Dagstuhl, Germany. Retrieved from http://inex.is.informatik.uni-duisburg.de:2003/

Hubscher, R., & Puntambekar, S. (2001). Navigation support for learners in hypertext systems: Is more indeed better? In J.D. Moore, C.L. Redfield, & W.L. Johnson (Eds.), Artificial intelligence in education: AI-ED in the wired and wireless future (pp. 13-20). Burke, VA: IOS Press.

Iksal S. (2002). Specification declarative et composition semantique pour des documents virtuels personnalisables. Doctoral thesis, ENST-Bretagne, December, 2002.

Jonassen, D., & Churchill, D. (2004). Is there a learning orientation in learning objects? International Journal of E-Learning, 3(2), 32-41.

Kabel, S., de Hoog, R., Wielinga, B., & Anjewierden, A. (2003). The added value of task and ontology-based instructional markup for information retrieval. Journal of the American Society for Information Science and Technology (JASIST), 55(4), 348-362.

Knuth, D.E. (1992). Literate programming. CLSI (Center for the Study of Language and Information) Lecture Notes Number 27, Leland Stanford Junior University.

Milosavljevic, M. (1997). Augmenting the user's knowledge via comparison. Proceedings of the 6th International Conference on User Modelling. Sardinia, Italy.

Mizoguchi, R., & Bourdeau, J. (2000). Using ontological engineering to overcome common AI-ED problems. International Journal of Artificial Intelligence in Education, 11(2), 107-121.

Shi, H., Rodriguez, O, Chen, S., & Shang, Y. (2004). Open learning objects as an intelligent way of organizing educational material. International Journal on E-Learning, 3(2), 51-63.

Suss, C., Freitag, B., & Brossler (1999, November). Metamodeling for web-based teachware management. In P.P Chen, D.W. Embley, J. Kouloumdijan, S.W. Liddle, & J.F. Roddick (Eds.), Proceedings of the International Workshop on the World-Wide Web and Conceptual Modeling (WWWCM'99) in conjunction with ER'99. Paris, France. LNCS 1727. Paris, France: Springer Verlag.

Vercoustre, A-M., & Paradis, P. (1997, November). A descriptive language for information object reuse through virtual documents. Proceedings of the 4th International, Conference on Object-Oriented Information Systems (OOIS'97), (pp. 299-311). Brisbane, Australia.

Wilkinson, R., Lu, S., Paradis, F., Paris C., Wan, S., & Wu, M. (2000, August). Generating personal travel guides from discourse plans. Proceedings of International Conference on Adaptive Hypermedia and Adaptive Web-based Systems. Trento, Italy.

ANNE-MARIE VERCOUSTRE

INRIA, France

[e-mail]

ALISTAIR MCLEAN

CSIRO-ICT Centre, Australia

[e-mail]