Adaptive learning object selection in intelligent learning systems.

Adaptive learning object selection and sequencing is recognized as among the most interesting research questions in intelligent web-based education. In most intelligent learning systems that incorporate course sequencing techniques, learning object selection is based on a set of teaching rules according to the cognitive style or learning preferences of the learners. In spite of the fact that most of these rules are generic (i.e., domain independent), there are no well-defined and commonly accepted rules on how the learning objects should be selected and how they should be sequenced to make "instructional sense." Moreover, to design highly adaptive learning systems a huge set of rules is required, since dependencies between educational characteristics of learning objects and learners are rather complex. In this article, we address the learning object selection problem in intelligent learning systems proposing a methodology that instead of forcing an instructional designer to manually define the set of selection rules, it produces a decision model that mimics the way the designer decides, based on the observation of the designer's reaction over a small-scale learning object selection case.

**********

The high rate of evolution of e-learning platforms implies that on the one hand, increasingly complex and dynamic web-based learning infrastructures need to be managed more efficiently, and on the other hand, new types of learning services and mechanisms need to be developed and provided. To meet the current needs, such services should satisfy a diverse range of requirements, for example, personalization and adaptation (Dolog, Henze, Nejdl, & Sintek, 2004; Vasilakos, Devedzic, Kinshuk, & Pedrycz, 2004). The field of computational intelligence in web-based education can contribute towards providing web-based technologies, methods, and techniques for supporting teaching and learning in an intelligent way.

Learning object selection is the first step to adaptive navigation and adaptive course sequencing. Adaptive navigation seeks to present the learning objects associated with an online course in an optimized order, where the optimization criteria takes into consideration the learner's background and performance on related learning objects (Brusilovsky, 1999), whereas adaptive course sequencing is defined as the process that selects learning objects from a digital repository and sequences them in a way, which is appropriate for the targeted learning community or individuals (Knolmayer, 2003). Selection and sequencing is recognized as among the most interesting research questions in intelligent web-based education (McCalla, 2000; Dolog & Nejdl, 2003; Devedzic, 2003).

Although many types of intelligent learning systems are available, we can identify five key components which are common in most systems, namely, the student model, the expert model, the pedagogical module, the domain knowledge module, and the communication model. Figure 1 provides a view of the interactions between these modules.

In most intelligent learning systems that incorporate course sequencing techniques, the pedagogical module is responsible for setting the principles of content selection and instructional planning. The selection of content (in our case, learning objects) is based on a set of teaching rules according to the cognitive style or learning preferences of the learners (Brusilovsky & Vassileva, 2003; Stash & De Bra, 2004). In spite of the fact that most of these rules are generic (i.e., domain independent), there are no well-defined and commonly accepted rules on how the learning objects should be selected and how they should be sequenced to make "instructional sense" (Knolmayer, 2003; Mohan, Greer, & McGalla, 2003). Moreover, to design highly adaptive learning systems a huge set of rules is required, since dependencies between educational characteristics of learning objects and learners' characteristics are rather complex.

[FIGURE 1 OMITTED]

In this article, we address the learning object selection problem in intelligent learning systems proposing a methodology that instead of forcing an instructional designer to manually define the set of selection rules; produces a decision model that mimics the way the designer decides, based on the observation of the designer's reaction over a small-scale learning object selection problem.

In the next section we discuss the learning object selection process as part of automatic course sequencing. The third section discusses the filtering process of learning objects used for reduction of learning objects searching space and proposes metadata elements that can be used for learning object filtering. The fourth section presents a methodology for capturing expert's decision model on learning objects selection and it constitutes the main contribution of this article. Finally, we present experimental results of the proposed methodology by comparing the resulting learning objects selected by the proposed method with those selected by experts.

LEARNING OBJECT SELECTION IN AUTOMATIC COURSE SEQUENCING

In automatic course sequencing, the main idea is to generate a course suited to the needs of the learners. As described in the literature, two main approaches for automatic course sequencing have been identified (Brusilovsky & Vassileva, 2003): Adaptive Courseware Generation and Dynamic Courseware Generation.

In Adaptive Courseware Generation, the goal is to generate an individualized course taking into account specific learning goals, as well as, the initial level of the student's knowledge. The entire course is adaptively generated before presenting it to the learner, instead of generating a course incrementally, as in a traditional sequencing context. In Dynamic Courseware Generation, the system observes the student progress during his interaction with the course and dynamically adapts the course according to the specific student needs and requirements. If the student's performance does not meet the expectations, the course is dynamically re-planned. The benefit of this approach is that it applies as much adaptivity to an individual learner as possible.

Both these techniques employ a prefiltering mechanism to generate a pool of learning objects that match the general content requirements. This pool can be generated from both distributed and local learning object repositories, provided that the appropriate access controls have been granted. The filtering process is based on general requirements such as characteristics of the language or the media of the targeted learning objects, as well as, on the use of ontologies for the domain in question (Domain Knowledge module). The result of the filtering process falls in a virtual pool of learning objects that will act as an input space for the content selector.

After the creation of the initial pool of learning objects, the content selection process is applied based on learner characteristics such as accessibility and competency characteristics or even historical information about related learning activities, included in the Student Model module. Figure 2 presents a generalized framework of the previously mentioned course sequencing techniques that use filtering, content selection, and instructional planning processes. In the next sections we will present some filtering elements based on the IEEE P1484.12.1 Learning Object Metadata (LOM) standard and we will analyze the methodology we propose for the content selection phase of automatic course sequencing.

LEARNING OBJECT FILTERING

The main goal of filtering is the reduction of the searching space. Learning Object Repositories often contain hundreds or thousands of learning objects, thus the selection process may require a significant computational time and effort. In most intelligent learning systems, learning object filtering is based either on the knowledge domain they cover or on the media type characteristics they contain (Kinshuk, Oppermann, Patel, & Kashihare, 1999). In the IEEE LOM metadata model, there exist a number of elements covering requirements such as the subject, the language and the media type of the targeted learning object. Table 1 presents the IEEE LOM elements we have identified for each one of the filtering categories and the conditions required.

Alternatively, filtering can be based on integration of the IEEE LOM metadata model elements and ontologies (Kay, Holden, 2002; Urban & Barriocanal, 2003), but those approaches assume that both the domain model and the learning objects themselves use the same ontology (Mohan et al., 2003) and limit the filtering only to knowledge domain filtering.

[FIGURE 2 OMITTED]

LEARNING OBJECT SELECTION

Typically, the design of highly adaptive learning systems requires a huge set of rules, since dependencies between educational characteristics of learning objects and learners are rather complex. This complexity introduces several problems on the definition of the rules required (Wu & De Bra, 2001; Calvi & Cristea, 2002), namely:

* Inconsistency, when two or more rules are conflicting.

* Confluence, when two or more rules are equivalent.

* Insufficiency, when one or more rules required have not been defined.

The proposed methodology is based on an intelligent mechanism that tries to mimic an instructional designer's decision model on the selection of learning objects. To do so, we have designed a framework that attempts to construct a suitability function that maps learning object characteristics over learner characteristics and vice versa.

The main advantage of this method is that it requires less effort by the instructional designer, since instead of identifying a huge set of rules, only the designer's selection from a small set of learning objects over a reference set of learners is needed. The machine learning technique will then try to discover the dependence between a learning object and learner characteristics that produce the same selection of learning objects per learner as the instructional designer did.

The proposed methodology does not depend on the characteristics used for learning objects and learner modeling, thus can be used for extraction of even complex pedagogy-related dependences. It is obvious that since characteristics/requirements like the domain are used for filtering, the dependencies produced are quite generic, depending only on the educational characteristics of the content and the cognitive characteristics of the learner.

Figure 3 presents a graphical representation of the Selection Model Extraction Framework, consisting of three main steps:

[FIGURE 3 OMITTED]

Step 1: Modeling and Selection of Criteria

The selection methodology is generic, independent of the learning object and the learner characteristics used for the selection. In our experiment, we used learning object characteristics derived from the IEEE LOM standard and learner characteristics derived from the IMS Global Learning Consortium Inc. Learner Information Package (LIP) specification. In Table 2 and 3 we have identified the LOM and LIP characteristics respectively, that can be used as an input space (set of selection criteria) to the learning object selector.

There exist many criteria affecting the decision of learning objects selection. Those criteria that lead to a straightforward exclusion of learning objects, such as the subject, the language, and the media type, are used for filtering. The rest of a set of criteria such as the educational characteristics of learning objects are used for selection model extraction, since the dependencies of those criteria can model the pedagogy applied by the instructional designer, when selecting learning objects.

Those criteria, due to the complexity of interdependencies between them, are the ones that cannot be directly mapped to rules from the instructional designer. Thus an automatic extraction method, like the proposed one, is needed.

Step 2: Selection Model Extraction

After identifying the set of characteristics/criteria (step 1) that will be used as the input space of the LO Selector, we try to extract for each learning object characteristic the expert's suitability evaluation model over a reference set of LIP-based characterized learners. The input to this phase is the IEEE LOM characteristics of a reference set of learning objects, the IMS LIP characteristics of a reference set of learners and the suitability preference of an expert for each of the learning objects over the whole reference set of learners. The model extraction methodology has the following formulation:

Let us consider a set of learning objects, called A, which is valued by a set of criteria g = ([g.sub.1], [g.sub.2],..., [g.sub.n]). The assessment model of the suitability of each learning object for a specific learner, leads to the aggregation of all criteria into a unique criterion that we call a suitability function: S(g) = S ([g.sub.1], [g.sub.2],...,[g.sub.n]). We define the suitability function as an additive function of the

form S(g) = [n.summation over (i=1)] [s.sub.i] ([g.sub.i]) with the following additional notation:

* [s.sub.i] ([g.sub.i]): Marginal suitability of the ith selection criterion valued [g.sub.i],

* S (g): Global suitability of a learning object.

The marginal suitability evaluation for the criterion [g.sub.i] is calculated using the formula [s.sub.i] (x) = [a.sub.i] + [b.sub.i]xexp(-[c.sub.i][x.sup.2]), where x is the corresponding value of the [g.sub.i] learning object selection criterion.

This formula produces, according to parameters a, b, and c as well as the value space of each criterion, the main criteria forms, we have identified:

* Monotonic form: when the marginal suitability of a criterion is a monotonic function;

* Non monotonic form: when the marginal suitability of a criterion is a non-monotonic function.

Figure 4 presents the different criteria forms supported by the proposed knowledge extraction methodology. The calculation of the optimal values of parameters a, b and c for each selection criterion is the subject of the Knowledge Model Extraction step.

Let us call P the strict preference relation and I the indifference relation. If [S.sub.o1] is the global suitability of a learning object [O.sub.1] and is the global suitability of a learning object [O.sub.2], then the following properties generally hold for the suitability function S:

[S.sub.o1] > [S.sub.o2] [left and right arrow] ([O.sub.1]) P ([O.sub.2]),

and the relation R = P [union] I is a weak order relation.

[S.sub.o1] > [S.sub.o2] <^=^> ([O.sub.1]) I ([O.sub.2]).

The expert's requested information then consists of the weak order R defined on A for several learner instances. Using the provided weak order relation R and based on the form definition of each learning object characteristic we can define the suitability differences [DELTA] = ([[DELTA].sub.1], [[DELTA].sub.2],...,[[DELTA].sub.m-1]), where m is the number of learning objects in the reference set A and [[DELTA].sub.k] = [S.sub.ok] - [S.sub.ok+1] [greater than or equal to] 0 depending on the suitability relation of (k) and (k+1) preferred learning object for a specific learner of the reference set.

[FIGURE 4 OMITTED]

We can introduce an error function e for each suitability difference: [[DELTA].sub.k] = [S.sub.ok] - [S.sub.ok+1] + [e.sub.k] [greater than or equal to] 0.

Using constrained optimization techniques, we can then solve the nonlinear problem:

Minimize [m-1.summation over (j=1)] ([e.sub.j])[.sup.2]

Subject to the constraints:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] for each one of the learners of the reference set.

This optimization problem will lead to the calculation of the optimal values of the parameter a, b and c for each learning object selection criteria over the reference set of learners.

Figure 5 presents the introduced error function, the suitability overestimation error as well as the suitability underestimation error e, on the ordinal regression curve, which is the suitability ranking of the reference set of learning objects versus the approximation of the global suitability of each one of the learning objects in the reference set.

Figure 6 presents a paradigm of marginal suitability extraction result (real expert's model and the resulted approximation), when using Interactivity Type, Interactivity Level, Semantic Density and Difficulty as LO selection characteristics for a specific learner.

[FIGURE 5 OMITTED]

[FIGURE 6 OMITTED]

Step 3: Extrapolation 

The purpose of this phase is to generalize the resulted marginal suitability model from the reference set of learners to all learners, by calculating the corresponding marginal suitability values for every combination of learner characteristics. This calculation is based on the interpolation of the marginal suitability values between the two closest instances of the reference set of learners.

Suppose that we have calculated the marginal suitability [s.sub.i.sup.[L.sub.1]] and [s.sub.i.sup.[L.sub.2]] of a criterion [c.sub.i] matching the characteristics of learners [L.sub.1] and [L.sub.2] respectively. We can then calculate the corresponding marginal suitability value for another learner L using interpolation if the characteristics of learner L are mapped inside the polyhedron that the characteristics of learners [L.sub.1] and [L.sub.2] define, using the formula:

[s.sub.i]([c.sub.i.sup.L]) = [s.sub.i]([c.sub.i.sup.[L.sub.1]]) + [[c.sub.i.sup.L] - [c.sub.i.sup.[L.sub.1]]]/[[c.sub.i.sup.[L.sub.2]] - [c.sub.i.sup.[L.sub.1]]] [[s.sub.i]([c.sub.i.sup.[L.sub.2]) - [s.sub.i]([c.sub.i.sup.[L.sub.1]])] if [s.sub.i]([c.sub.i.sup.[L.sub.2]]) > [s.sub.i]([c.sub.i.sup.[L.sub.1]])

Figure 7 presents the marginal suitability values of Interactivity Type criterion for different LIP Instances and Figure 8 the calculation of the corresponding marginal suitability value for a new (external from the reference set) learner. Due to the high dimensional space of the marginal suitability surface, we represent examples when only one characteristic is used for learner modeling.

[FIGURE 7 OMITTED]

[FIGURE 8 OMITTED]

Let [C.sub.i] = [[c.sub.i*], [c.sub.i]*]i = 1, 2,... n be the intervals in which the values of each criterion--for both learning object and learners--are found, then we call global suitability surface the space C = [x.sup.n.sub.i=1] [C.sub.i]. The calculation of the global suitability over the previously mentioned space is the addition of the marginal suitability surfaces for each of the learning object characteristics over the whole combination set of learner characteristics.

Figure 9 presents an example of marginal suitability surface and Figure 10 the corresponding global suitability surface, after the summation of the marginal surfaces for each one of the learning object characteristics. Due to the high dimensional space of these surfaces, we represent them only in the case when only one characteristic is used for learner modeling.

EXPERIMENTAL RESULTS AND DISCUSSION

To evaluate the total efficiency of the proposed methodology both on calculating the suitability on the training set of learning objects and on estimating the suitability of learning objects external from the reference set, we have designed an evaluation criterion, defined by:

Success (%) = 100* [[Correct Learning Objects Selected]/n]

[FIGURE 9 OMITTED]

[FIGURE 10 OMITTED]

where n is the number of the desired learning objects from the virtual pool that will act as input to the instructional planner. We assume that the number of desired learning objects is less than the total number of learning objects in the input space (learning objects pool) and that both the learning object metadata and the learner information metadata have normal distribution over the value space of each criterion.

Additionally, we have classified the learning objects, for both testing and estimation set, in two classes according to their aggregation level, since granularity is a parameter affecting the capability of an instructional designer to select learning content for a specific learner. The classification is based on the value space of the "General/Aggregation_Level" element of the IEEE LOM standard. Table 4 presents a description of the two classes used.

We present experimental results of the proposed methodology by comparing the resulting selected learning objects with those selected by experts. We have evaluated the success on both the training set of learning objects (Training Success) and on the suitability estimation of learning objects external from the reference set (Estimation Success). Figure 11 and 12 present average experimental results for learning objects with aggregation level 1 and 2 respectively.

If we consider that for one learner instance, the different combinations of learning objects, calculated as the multiplication of the value instances of characteristics presented in Table 2, lead to more than 900,000 learning objects, it is evident that it is almost unrealistic to assume that an instructional designer can manually define the full set of selection rules which correspond to the dependencies extracted by the proposed method and at the same time to avoid the inconsistencies, confluence and insufficiency of the produced selection rules.

The proposed methodology is capable of effectively extracting dependencies between learning object and learner characteristics affecting the decision of an instructional designer on the learning object selection problem.

More analysis of the results, presented in Figures 11 and 12, shows that when the desired number of learning objects (n) is relatively small (less than 100), the selected learning objects by the extracted decision model are almost similar to those the instructional designer would select.

On the other hand, when the desired number of learning objects is relatively large (about 500) the success of the selection is affected, but remains at acceptable level (about 90%).

Another parameter affecting the selection success is proved to be the granularity of learning objects. Granularity mainly affects the capability of an instructional designer to express selection preferences over learning objects. Learning objects with small aggregation level have bigger possibility of producing "gray" decision areas, where the instructional designer cannot decide which learning object matches most the cognitive style or learning preferences of a learner. In our experiments, learning objects with aggregation level 2, which can be small or even bigger collections of learning objects with aggregation level 1, appear to have less possibility of producing indifference relations, enabling to make secure decisions even for bigger desired number of learning objects (n=200).

CONCLUSIONS AND FUTURE RESEARCH

In this article we address the learning object selection problem in intelligent learning systems proposing a methodology that instead of forcing an instructional designer to manually define the set of selection rules; produces a decision model that mimics the way the designer decides, based on the observation of the designer's reaction over a small-scale learning object selection problem.

Since one of the primary design goals of learning objects is reusability in a variety of diverse learning contexts, learning objects are generally designed in a highly de-contextualized manner (South & Monson, 2000). At the same time, it is nearly impossible to define learning characteristics, like difficulty or semantic density, which affect both selection and sequencing of learning objects.

The proposed content selection methodology can provide the framework for designing highly adaptive learning systems, provided that learning objects are as small as needed--learning threshold--for a content author to be able to identify the pedagogical features they contain (Gibbons, Nelson, & Richards, 2000). Figure 13 presents the optimum granularity of learning objects as the space in between the learning and the context thresholds.

[FIGURE 13 OMITTED]

Future research includes learning object decomposition from existing courses, allowing reuse of the disaggregated learning objects in different educational contexts. The intelligent selection of the disaggregation level and the automatic structuring of the atoms (raw media) inside the disaggregated components to preserve the educational characteristics they were initially designed for, is a key issue in the research agenda for learning objects (Duval, 2003).

Table 1 LOM Elements and Conditions for Learning Object Filtering

Filters   IEEE LOM Path    Explanation                   Usage Condition

Subject   LOM/General/     A keyword or phrase
          Keyword          describing the topic of a
                           Learning Object
          LOM/General/     The time, culture, geography
          Coverage         or region to which a
                           Learning Object applies.
          LOM/             This category describes       LOM/
          Classification   where a Learning Object       Classification/
                           falls within a particular     Purpose =
                           classification system.        "Discipline" or
                                                         "Idea"
Language  LOM/General/     The primary human language/s
          Language         used within a Learning
                           Object.
          LOM/             The human language used by
          Educational/     the typical intended user of
          Language         a Learning Object
Media     LOM/Technical/   Technical data type/s of all
          Format           the components of a Learning
                           Object
          LOM/Technical/   The size of the digital
          Size             Learning Object in bytes.
                           This element refers to the
                           uncompressed size.
          LOM/Technical/   Time a continuous Learning
          Duration         Object takes when played at
                           intended speed.
          LOM/Lifecycle/   The completion status or      LOM/
          Status           condition of a Learning       Lifecycle/
                           Object                        Status!=
                                                         "unavailable"
          LOM/Rights/      Whether use of a Learning
          Cost             Object requires some kind of
                           payment.

Table 2 LO Selector Input Space (Learning Object characteristics)

Selection
Criteria     IEEE LOM Path            Explanation

General      LOM/General/Structure    Underlying organizational
                                      structure of a Learning Object
             LOM/General/Aggregation  The functional granularity (level
             Level                    of aggregation) of a Learning
                                      Object.
Educational  LOM/Educational/         Predominant mode of learning
             Interactivity Type       supported by a Learning Object
             LOM/Educational/         The degree to which a learner can
             Interactivity Level      influence the aspect or behavior
                                      of a Learning Object.
             LOM/Educational/         The degree of conciseness of a
             Semantic Density         Learning Object, estimated in
                                      terms of its size, span or
                                      duration.
             LOM/Educational/         Age of the typical intended user.
             Typical Age Range        This element refers to
                                      developmental age and not
                                      chronological age.
             LOM/Educational/         How hard it is to work with or
             Difficulty               through a Learning Object for the
                                      typical intended target audience.
             LOM/Educational/         Principal user(s) for which a
             Intended End User Role   Learning Object was designed, most
                                      dominant first.
             LOM/Educational/         The principal environment within
             Context                  which the learning and use of a LO
                                      is intended to take place.
             LOM/Educational/         Typical time it takes to work with
             Typical Learning Time    or through a LO for the typical
                                      intended target audience.
             LOM/Educational/         Specific kind of Learning Object.
             Learning Resource Type   The most dominant kind shall be
                                      first.

Table 3 LO Selector Input Space (Learner characteristics)

Selection       IMS LIP Path        Explanation          Usage Condition

Accessibility   LIP/Accessibility/  The type of          --
                Preference/         cognitive
                typename            preference
                LIP/Accessibility/  The coding assigned  --
                Preference/         to the preference
                prefcode
                LIP/Accessibility/  The type of          --
                Eligibility/        eligibility being
                typename            defined
                LIP/Accessibility/  The type of          --
                Disability/         disability being
                typename            defined
Qualifications  LIP/QCL/Level       The level/grade of   LIP/QCL/
Certifications                      the QCL              Typename, LIP/
Licenses                                                 QCL/Title and
                                                         LIP/QCL/
                                                         Organization
                                                         should refer to
                                                         a qualification
                                                         related with
                                                         the objectives
                                                         of the learning
                                                         goal
                                                         LIP/QCL/date >
                                                         Threshold
Activity        LIP/Activity/       The number of        LIP/Activity/
                Evaluation/         attempts made on     Typename,
                noofattempts        the evaluation.      LIP/Activity/
                                                         status, LIP/
                                                         Activity/
                                                         units and
                                                         LIP/Activity/
                                                         Evaluation/
                LIP/Activity/       Information that     Typename should
                Evaluation/         describes the        refer to a
                result/             scoring data.        qualification
                LIP/Activity/       the scoring data     related with
                Evaluation/         itself.              the
                result/score                             objectives of
                interpretscope                           the learning
                                                         goal
                                                         LIP/Activity/
                                                         date >
                                                         Threshold
                                                         LIP/Activity/
                                                         Evaluation/
                                                         date >
                                                         Threshold

Table 4 Learning Objects Aggregation Level according to IEEE LOM
standard

IEEE LOM Element           Value Space  Description

General/Aggregation_Level       1       The smallest level of
                                          aggregation, e.g. raw media
                                          data or fragments
                                2       A collection of level 1 learning
                                          objects, e.g. a lesson chapter
                                          or a full lesson

Aggregation Level 1

                10   20     50    100   200   500

Training Set    100  100    96.7  95.4  92.1  90.6
Estimation Set  100   99.2  95.3  93.1  90.6  88.4

Figure 11. Average experimental results for learning objects with
aggregation Level 1

Aggregation Level 2

                10   20   50    100   200   500

Training Set    100  100  98.3  97.1  95.6  93.4
Estimation Set  100  100  96.5  94.8  92.3  90.8

Figure 12. Average experimental results for learning objects with
aggregation Level 2

Acknowledgements

The work presented in this article is partially supported by the European Commission under the Information Society Technologies (IST) programme of the 6th FP for RTD--project ICLASS contract IST-507922.

References

Brusilovsky, P. (1999). Adaptive and intelligent technologies for web-based education. Kunstliche Intelligenz, 4, 19-25.

Brusilovsky, P., & Vassileva, J. (2003). Course sequencing techniques for large-scale web-based education. International Journal of Continuing Engineering Education and Life-long Learning, 13(1/2), 75-94.

Calvi, L., & Cristea, A. (2002). Towards generic adaptive systems: Analysis of a case study. In Proceedings of the 2nd International Conference on Adaptive Hypermedia and Adaptive Web Based Systems, Malaga, Spain.

Devedzic V.B. (2003). Key issues in next-generation web-based education. IEEE Transactions on Systems, Man and Cybernetics Part C, 33(3), 339-349.

Dolog P., & Nejdl, W. (2003). Challenges and benefits of the semantic web for user modeling. Workshop on adaptive hypermedia and adaptive web-based systems. In Proceedings of the 12th International World Wide Web Conference, Budapest, Hungary.

Dolog P., Henze, N., Nejdl, W., & Sintek, M. (2004). Personalization in distributed elearning environments. In Proceedings of the 13th International World Wide Web Conference, New York, USA.

Duval E., & Hodgins, W. (2003). A LOM research agenda. In Proceedings of the 12th International World Wide Web Conference, Budapest, Hungary.

Gibbons A. S., Nelson, J., & Richards, R. (2000). The nature and origin of instructional objects. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version. [Online]. Available: http://reusability.org/read/chapters/gibbons.doc

Kay, J., & Holden, S. (2002). Automatic extraction of ontologies from teaching document metadata. ICCE workshop on concepts and ontologies in web-based educational systems. In Proceedings of the International Conference on Computers in Education, Auckland, New Zealand.

Kinshuk, Oppermann, R., Patel, A., & Kashihara, A. (1999). Multiple representation approach in multimedia-based intelligent educational systems. In S.P. Lajoie & M. Vivet (Eds.), Artificial intelligence in education, (pp. 259-266). Amsterdam, The Netherlands: IOS Press.

Knolmayer, G.F. (2003). Decision support models for composing and navigating through e-learning objects. In Proceedings of the 36th IEEE Annual Hawaii International Conference on System Sciences, Hawaii, USA.

McCalla, G. (2000). The fragmentation of culture, learning, teaching and technology: Implications for the artificial intelligence in education research agenda in 2010. International Journal of Artificial Intelligence in Education, 11, 177-196.

Mohan, P., Greer, J., & McGalla, G. (2003). Instructional planning with learning objects. Workshop on knowledge representation and automated reasoning for e-learning systems. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, Acapulco, Mexico.

South, J.B., & Monson, D.W. (2000). A university-wide system for creating, capturing, and delivering learning objects. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version, [Online]. Available: http://reusability.org/read/chapters/south.doc

Stash, N., & De Bra, P. (2004). Incorporating cognitive styles in AHA! First international workshop on adaptable and adaptive hypermedia. In Proceedings of the IASTED Conference on Web Based Education, Innsbruck, Austria.

Urban, M.S., & Barriocanal, E.G. (2003). On the integration of IEEE-LOM metadata instances and ontologies. Learning Technology Newsletter, 5(1). [Online]. Available: http://lttf.ieee.org/learn_tech/issues/january2003/

Vasilakos, T., Devedzic, V., Kinshuk, & Pedrycz, W. (2004). Computational intelligence in web-based education. Journal of Interactive Learning Research, 15(4), 299-318. Special Issue.

Wu, H., & De Bra, P. (2001). Sufficient conditions for well-behaved adaptive hypermedia systems. In Proceedings of the First Asia-Pacific Conference on Web Intelligence: Research and Development, Maebashi City, Japan.

PYTHAGORAS KARAMPIPERIS AND DEMETRIOS SAMPSON

University of Piraeus, Piraeus, Greece; and Informatics and Telematics Institute, Hellas, Greece

[e-mail]

[e-mail]