Printer Friendly

New MCEQLS TFN method for evaluating quality and reusability of learning objects.


The aim of the paper is to investigate, propose, and demonstrate examples of practical application of MCEQLS (Multiple Criteria Evaluation of the Quality of Learning Software) TFN (Trapezoidal Fuzzy Numbers) method for the expert evaluation of the quality and reusability of learning objects (LOs). This problem is of very high practical importance for the educational sector (both Universities and schools) that needs exact, clear, simple to use and efficient models and methods for evaluating the quality and reusability of LOs in the market, both proprietary and open ones. These proper quality evaluation models and methods have to meet all the educational stakeholders' (i.e. educational institutions, policy makers, content creators / publishers, researchers, and practitioners (teachers)) needs. The authors have created a coherent, simple to use and efficient model and methods for evaluating the quality and reusability of LOs for strategic EU-funded eQNet project. 9 European Ministries of Education directly participate in eQNet, and the other European Network's (EUN 2012) Ministries use the results of the project in their countries.

The various approaches to LOs attempt to meet two common objectives: to reduce the overall costs of LOs, and to obtain better LOs. In this sense, we create cost-benefit economic utility approach to using educational materials all over the world. Both these objectives agree with the notion of LOs reusability. LOs reusability is one of the main conditions to achieve high LOs effectiveness and efficiency level. The need for reusability of LOs has at least three elements: Interoperability (i.e. LO is interoperable and can be used in different platforms); Flexibility in terms of pedagogic situations (i.e. LO can fit into a variety of pedagogic situations); and Modifiability to suit particular teacher's or student's needs (i.e. LO can be made more appropriate to different pedagogic situations by modifying it to suit particular teacher's or student's needs) (Kurilovas, Dagiene 2010). Reusability of LOs (or their ability to "travel well" between different contexts and education systems) is considered by the authors as a part of the overall quality of LOs. This means that any high quality LO has some reusability level (or potential to "travel well"), but this does not mean that any reusable LO is qualitative one.

The rest of the paper is organised as follows: overview of the used notions, methods, and related works in the area is presented in Section 1; a new MCEQLS FTN method is presented in Section 2; MCEQLS TFN application research results are presented in Section 3; discussion is presented in Section 4, and conclusions are presented in the last section. Section 3 is divided into two separate parts--(1) creation of comprehensive and coherent LOs quality model (criteria system), and (2) experimental application of MCEQLS TFN method for evaluating the quality of several real-life LOs alternatives used in eQNet project.

1. Overview of the used notions, methods and related works in the area

1.1. Learning objects and their reusability

Learning objects are the elements of a new type of computer-based instruction grounded in the object-oriented paradigm of computer science. Object-orientation highly values the creation of components (called "objects") that can be reused in multiple contexts. This is the fundamental idea behind LOs: instructional designers can build small (relative to the size of an entire course) instructional components that can be reused a number of times in different learning contexts. Additionally, LOs are generally understood to be digital entities deliverable over the Internet and any number of people can access and use them simultaneously. Moreover, those who incorporate LOs can collaborate on and benefit immediately from the new versions. Supporting the notion of small, reusable chunks of instructional media, Reigeluth and Nelson (1997) suggest that when teachers first gain access to the instructional materials, they often break the materials down into their constituent parts. They then reassemble these parts in the ways that support their individual instructional goals. This suggests one reason why reusable instructional components (i.e. LOs) may provide instructional benefits: if instructors received the instructional resources as the individual components, this initial step of decomposition could be bypassed, potentially increasing the speed and efficiency of instructional development.

The Learning Technology Standards Committee of the Institute of Electrical and Electronics Engineers chose the term "learning objects" to describe these small instructional components, and has established a working group and provided a working definition: "Learning Objects are defined here as any entity, digital or non-digital, which can be used, reused or referenced during technology supported learning" (IEEE LTSC 2012). Examples of LOs can include multimedia content, instructional content, learning objectives, instructional software and software tools, as well as persons, organisations or events referenced during technology supported learning.

According to Wiley (2000), this is a very wide definition that can be interpreted to include just about anything, which makes the definition somewhat hard to use. This definition is extremely broad, and upon examination fails to exclude any person, place, thing, or idea that has existed at anytime in the history, since any of these could be "referenced during technology supported learning". Therefore, Wiley (2000) defined a learning object as "any digital resource that can be reused to support learning".

This definition of LO is proposed for two reasons. First, the definition is sufficiently narrow to define a reasonably homogeneous set of things: reusable digital resources. Second, the proposed definition is based on the LTSC definition (and defines a proper subset of learning objects as defined by the LTSC), making issues of compatibility of LO as defined within Wiley (2000) and LO as defined by the LTSC explicit. The proposed definition captures the critical attributes of a LO, "reusable", "digital", "resource", and "learning", as does the LTSC definition. With that compatibility made explicit, the proposed definition differs from the LTSC definition in two important ways: (1) the definition explicitly rejects non-digital and non-reusable resources. The definition also drops the phrase "technology supported" which is now implicit, because all LOs are digital; and (2) the phrase "to support" has been substituted in place of "during" in the LTSC definition. Use of an object "during" learning doesn't connect its use to learning. The definition emphasizes the purposeful use (by either an instructional designer, an instructor, or a student) of these objects to support learning (Wiley 2000).

This concept of LOs has gained wide spread acceptance in the world of e-learning. The main purpose of LOs is to provide a modularised model based on standards that enhance flexibility, platform independence, and reuse of learning content--as well as providing a higher degree of control for teachers and learners. The definition and meaning of the term "learning object" varies considerably between different actors and communities as well as over time. The idea of LOs is (and has always been) to organise digital learning content into small, fairly context-independent chunks that can be assembled, disassembled and combined in different ways and in different learning contexts. LOs from different vendors can be combined with each other to form modules that can be used in a specific learning context. Ideas that are much inspired by experience from system development which shows that the component-based approaches are favourable for quality and significantly reduce time to market (Brereton et al. 2000).

According to Paulsson and Naeve (2006a), depending on the approach, the management and composition of LOs into larger learning modules are managed by a teacher, a learner or someone outside the educational institution (e.g. by a content provider or someone distributing a learning package). However, much of this is still just a vision and there is an impending risk that the potential multi-billion Euro market for LOs will not develop as anticipated if interoperability and common concepts cannot be guaranteed.

The lack of common definitions and models for LOs is a threat to interoperability and technical quality--as well as a threat to the concept itself. McGreal (2004) points out, after a review of the LO terminology, that a LO may range "from anything to everything". Much of the vision surrounding the LOs concept has yet to be fulfilled.

An important condition in order to realise LOs is the use of standards for learning technology, such as IMS (e.g. IMS CC, IMS CP), IEEE/LTSC LOM, SCORM and others. This is an important reason why the LO community to a large extent has set the learning technology standardisation agenda. Much of the standardisation work focuses on descriptive information (so-called metadata), structure and packaging of learning content and not so much on the learning architecture and virtual learning environments (VLEs). However, since many of the content-centred standards have matured, the focus is slowly changing and the VLE is receiving more attention.

Sosteric and Hesemeier (2002) define LOs as digital objects that have a formal educational purpose within a predestined pedagogical context. They take on a rather traditional view on learning. There is a risk that such view can limit the pedagogical choices as well as the innovative aspects of using information and communication technologies and digital learning content. McGreal (2004) means that the LO definition should be limited down to units that practitioners already prefer to work with and suggests a definition where LO is "any reusable digital resource that is encapsulated in a lesson or assemblage of lessons grouped in units, modules, courses and even programmes". Song et al. (1999) take a slightly different approach to LOs as they mean that LOs should be regarded as decomposable, and that there must be a separation between data, operations and the carrier of the data. They also argue that a LO should be described using a set of attributes and relationships to other LOs. Song et al. (1999) focus mainly on the internal structure of LOs and their relations to other LOs. Their approach rests heavily on experience and concepts from object-oriented programming. Paulsson and Naeve (2006b) suggest a model and a taxonomy that is compatible with the taxonomy suggested by Wiley (2002) and in some ways architecturally similar to the model suggested by Song et al. (1999). The basic idea of this taxonomy is to separate the data, application logics and presentation of a LO. This is accomplished by the introduction of three different types of components, i.e. data-objects (referred to as "fundamental LOs" by Wiley (2002)), "raw assets" by Koppi and Lavitt (2003) and "information objects" by McGreal (2004), and resource objects. Resource objects are of two types: "helper resource objects" and "creator resource objects". The first type is used to add application logics and / or presentation to a data-object and the second is used to add application logics that add to the learning environment without necessary being tied to specific content.

According to Paulsson and Naeve (2006a), currently, "reuse" of LOs is often limited to packaging and sequencing of the pre-packaged units, which have nothing else in common other than that they are called LOs, are described using the educational metadata, and are packaged and sequenced together. They can however not function together in terms of interacting, exchanging data and messages, being subordinate to the same look and feel, or share application logics. None of their studied LOs used any standard interfaces for interaction.

1.2. Multiple criteria decision analysis in e-learning science

According to Oliver (2000), evaluation can be characterised as "the process by which people make judgements about value and worth". In the context of learning technology, this judgement process is complex and often controversial. Although the notion of evaluation is rooted in a relatively simple concept, the process of judging the value of learning technology is complex and challenging. Quality evaluation is referred here as "the systematic examination of the extent to which an entity (part, product, service or organisation) is capable of meeting specified requirements" (ISO/IEC 14598-1:1999). Expert evaluation is referred here as a multiple criteria evaluation of learning software aimed at the selection of the best alternative based on score-ranking results (Kurilovas, Dagiene 2010).

Antucheviciene et al. (2006) believe that in everyday problems a decision maker often faces interactive and interrelated criteria. According to Petkus et al. (2009), decision making can be classified as (1) multiple attribute decision making for the sorting or ranking of alternatives according to several attributes (Turskis 2008; Zavadskas et al. 2006); (2) multiple criteria decision making (MCDM), for driving a vector optimisation based design process to a solution. The author's proposed method fits the last approach.

During the last year, MCDM related research results were presented in numerous papers (e.g. Balezentis et al. 2012; Brauers, Zavadskas 2012; Liu 2012; Stanujkic et al. 2012; Zolfani et al. 2012).

The practical problem analysed in the paper is how to choose the best reusable LO alternative in the market or create it. Here "the best" alternative means an alternative of the highest quality. Therefore, the main scientific problem analysed in the paper is creation of simple to use and efficient models and methods for the expert evaluation of the quality and reusability of LOs. According to Zilinskas (2005), these two characteristics are the most important while deciding on possible application of different multiple criteria decision analysis (MCDA) methods. The efficiency of the method applied is the most important characteristic of the method from the point of view of the methods' creators. This characteristic could be estimated in experimental way, and one could decide on the practical efficiency of the method only when it is widely used and acceptable for many different users (Zilinskas 2005).

Each alternative in a multiple criteria decision-making problem can be described by a set of criteria that can be qualitative and quantitative. There are many methods for determining the ranking of a set of alternatives in terms of a set of decision criteria. In a multiple criteria approach, the experts seek to build a set of criteria using several points of view. One of the main problems in this task is how to establish a "proper" (i.e. as objective as possible) system of LOs quality criteria which should reflect the objective scientific principles of constructing a model (criteria system) for their quality evaluation. These issues have been analysed in the research works on MCDA. Another problem is application of suitable MCDA methods in the numerical evaluation of the quality and reusability of LOs. The main problems of the existing approaches in the area are a high level of expert evaluation subjectivity as well as their insufficient exactness, clarity, usability and efficiency. Moreover, existing literature analysis performed by the authors using systematic review methodology showed that currently there are no simple to use and efficient models and methods proposed for the expert evaluation of LOs quality and reusability.

Therefore, in their previous works, the authors analysed several scientific methods, requirements and principles to minimise the aforementioned problems in evaluating the quality of learning software and proposed to use MCEQLS approach based on vector optimisation. MCEQLS approach, in their opinion, is quite easy-to-use and efficient in real-life situations. MCEQLS approach was presented in Kurilovas and Dagiene (2010) and refined in Kurilovas et al. (2011). In those works, it was shown that the MCEQLS approach could significantly improve the quality of the expert evaluation of learning software and noticeably reduce the expert evaluation's subjectivity level.

2. New MCEQLS TFN method in multi-criteria decision-making in e-learning science

2.1. MCEQLS approach

MCEQLS approach consists of the complex application of a number of scientific principles, methods and requirements as follows: (1) the principles of MCDA for identification of the quality criteria, (2) technological quality criteria classification principle, and (3) experts' additive utility function using normalised weights of the quality criteria (Kurilovas et al. 2011). A short description of all those stages of MCEQLS approach is as follows:

(1) According to Zavadskas and Turskis (2008), each alternative in the multi-criteria decision making problem can be described by a set of criteria. It can be qualitative and quantitative. According to Belton and Stewart (2002), in identifying criteria for the decision analysis, the following considerations (principles) are relevant to all MCDA approaches: value relevance; understandability; measurability; non-redundancy; judgmental independence; balancing completeness and conciseness; operationality; and simplicity versus complexity. These principles should be applied in creating a proper LO quality model.

(2) According to software quality standard (ISO/IEC 9126-1:2001(E)), we can divide the learning software quality criteria into "internal quality" and "quality in use" criteria. Internal quality is a descriptive characteristic that describes the quality of software independently from any particular context of its use, while Quality in use is evaluative characteristic of software obtained by making a judgment based on the criteria that determine the worthiness of software for a particular project.

(3) One of the suitable vector optimisation methods is the experts' additive utility function represented by equation (1) below including the learning software quality criteria, their ratings (values) and weights:

f (X) = [m.summation over (i=1)] [a.sub.i]([X.sub.i]). (1)

Here f ([X.sub.j]) is the rating (value) of the criterion i = 1, 2 ... m for the each of the examined software alternatives [X.sub.j], and [a.sub.i] are the weights of the evaluation criteria. The weight [a.sub.i] of the evaluation criterion reflects the expert's opinion on the criterion's importance level in comparison with the other criteria. The following 'normalisation' requirement (Eq. (2)) exists for the weights of the evaluation criteria in equation (1):

[m.summation over (i=1)][a.sub.i] = 1, [a.sub.i] > 0. (2)

According to Zavadskas and Turskis (2010), the normalisation aims at obtaining comparable scales of criteria values. The major is the meaning of the utility function in equation (1) the better the learning software meets the quality requirements in comparison with the ideal (100%) quality (Kurilovas et al. 2011). The biggest value of the function in equation (1) is the best, and the least one is the worst (Zavadskas, Turskis 2010).

Now one has to establish proper ratings (values) and weights of the quality criteria to apply equation (1).

2.2. New MCEQLS TFN Method

2.2.1. Method of TFN application to establish the values of the quality criteria

According to Ounaies et al. (2009), the wide-used measurement criteria of the decision attributes quality are mainly qualitative and subjective. In this context, decisions are often expressed in the natural language, and evaluators are unable to assign exact numerical values to different criteria.

Assessment can be often performed by the linguistic variables such as 'bad', 'poor', 'fair', 'good' and 'excellent'. These linguistic variables allow reasoning with imprecise information, and they are commonly called fuzzy values. Integrating these different judgments to obtain a final evaluation is not evident. In order to solve this problem Ounaies et al. (2009) suggested using the fuzzy group decision making theory to obtain final assessment measures. According to their proposal, first, linguistic variable values should be mapped into fuzzy numbers, and, second, into non-fuzzy values. In the present paper, the authors use trapezoidal fuzzy numbers (TFNs) for evaluating the quality and reusability of LOs alternatives (Fig. 1).


According to Kurilovas et al. (2011), in the case of using average TFNs, linguistic variables conversion into non-fuzzy values of the evaluation criteria should be as follows (Table 1).

2.2.2. Method of TFN application to establish the weights of the quality criteria

The authors propose to use TFN method not only for establishing the ratings (values) of the quality criteria but also for establishing the proper weights of the quality criteria. In the case of using average TFNs, linguistic variables conversion into non-fuzzy weights of the evaluation criteria should be as follows (Table 2).

If an expert evaluator establishes a weight of the criterion i in a form of a linguistic variable, we can convert it into trapezoidal fuzzy number [m.sup.i.sub.f]. According to Uppuluri (1989), if we have t experts (two in our case), we can calculate it using the experts' average (Eq. (3)):

[m.sup.i.sub.f] = 1/t [t.summation over (k=1)][m.sup.i.sub.k]. (3)

If we want to normalise the weights, we should use equation (4):

[a.sub.i] = [m.sup.i.sub.f]/[m.summation over (s=1)][m.sup.s.sub.f]. (4)

In this case, one could obtain normalised weights: [m.summation over (i=1)][a.sub.i] = 1.

According to MCEQLS approach, after using the TFN application method, the experts' additive utility function (Eq. (1)) should be applied to calculate the weights and the ratings (values) of the experts' additive utility function (Eq. (1)) for each of the explored LOs alternatives.

MCEQLS TFN method is convenient in the case when there are many probably qualitative alternatives in the market. It shows not only the difference between the quality of the evaluated alternatives expressed in per cent, but also the difference between the quality of the real-life alternatives and the ideal quality.

3. Experimental application of MCEQLS TFN method

3.1. Learning objects quality model

The proposed LO quality and reusability model (Table 3) includes three groups of criteria, namely, technological, pedagogical and intellectual property rights (IPR) criteria. The model consists of 10 quality criteria, 4 of them dealing with technological quality of LOs, 5--with their pedagogical quality, and 1--with IPR issues. Those criteria are based on the authors' previous works (Kurilovas et al. 2011; Kurilovas, Dagiene 2010) results and communication with eQNet partners resulted in LO quality criteria deliverables.

The authors' analysis has shown that the model presented fits all MCDA criteria identification principles proposed by Belton and Stewart (2002). Taking into account Non-redundancy, Judgmental independence, Balancing completeness and conciseness, Operationality, and Simplicity versus complexity MCDA criteria identification principles the authors consider that the following 10 LOs quality criteria should construct the comprehensive and coherent LOs quality model.

In order to create the proper model, the authors first of all have analysed a number of existing sets of quality criteria for evaluating the technological quality of LOs. According to this analysis, technological quality criteria and their sub-criteria should be as follows:

"Internal quality" criteria:

(1) Technological reusability (according to (Kurilovas, Dagiene 2010)): Interoperability (i.e. metadata accuracy; compliance with the main import/export standards); Decontextualisation (i.e. LO aggregation (granularity) level); LO modularity (i.e. full functionality of the parts of a content item on their own); Cultural and learning diversity (i.e. adaptability: LO flexibility (possibility to modify LO, e.g. from a configuration file, from a plain text file or because it is provided along with its source code or an authoring tool; LO suitability for localisation); Accessibility (i.e. design of controls and presentation formats to accommodate disabled learners, compliance with accessibility standards).

(2) Architecture: Is LO architecture layered in order to separate data, presentation and application logics?

(3) Robustness, technical stability (according to (Becta 2007)): Having help functions that identify common user problems and their solutions; Having navigational actions that can be undone; Giving quick, visible and audible responses to user actions; Allowing the user to exit at any point; Not being adversely affected by user experimentation and error. If users do experience an error they should be able to recover quickly and, where appropriate, be informed about the nature of the error.

"Quality in use" criterion:

(4) Design and usability (i.e. design of visual and auditory information for enhanced learning and efficient mental processing): aesthetics; navigation; user-friendly interface; information structuring; personalisation (Kurilovas, Dagiene 2010).

Those LOs technological quality criteria are included into a majority of the existing LOs quality models. 'Interoperability' and 'Accessibility' criteria being independent criteria in, e.g. Leacock, Nesbit (2007) or Becta (2007) are included as sub-criteria into 'Technical reusability' criterion in the presented model. There are several reasons for this, e.g. both 'Interoperability' and 'Accessibility' criteria deal with international interoperability standards and specifications, both influence LO technical reusability level in different repositories and platforms. MCDA Non-redundancy principle is applied here.

On the other hand, the authors have analysed a number of existing sets of quality criteria for evaluation of pedagogical quality of LOs (e.g. Becta 2007; Leacock, Nesbit 2007; MELT 2008; Vargo et al. 2003).

Suitable pedagogical criteria and their sub-criteria based on MCDA principles are as follows:

(5) Trans-national or multidisciplinary / cross-curricular topic (eQNet 2012).

(6) Interactivity, strong visual element (e.g. LOs include animations, images, short videos and simulations that are self-explanatory or have just a few text labels or icons/buttons for start, stop, etc.; strong visual structure) (MELT 2008).

(7) Language independence (i.e. LO is not text-heavy; LOs may have little or no text; or low language dependence (easily translatable); or LOs are multilingual, i.e. LOs have been designed to be language customisable and are already offered in more than one language) (MELT 2008).

(8) Ease of use, intuitiveness (i.e. users can find their way through the resource almost intuitively, they can broadly understand what is the intended learning objective or topic; LOs provide appropriate guidance, where necessary, for learners and/or practitioners; LOs make appropriate assumptions about the ICT skills of users, both learners and practitioners, or provide straightforward guidance on this (MELT 2008); and LOs not present a barrier or impede the learning experience) (Becta 2007).

(9) Additional methodological support for teachers is not needed (eQNet 2012). IPR quality criterion:

(10) Open license, free to use, open code: Licensing (clear rules, e.g. compliance with Creative Commons); Economic efficiency--Cost versus Quality taking into account probable LO reusability level (Kurilovas et al. 2011).

Now let us analyse the relationships between LO reusability elements (see Introduction) and LOs quality criteria presented in Table 3.

Every LO quality criterion in Fig. 2 fits one or more LO reusability elements. The details of interconnection of LOs modifiability element and LO quality criteria are elaborated as follows. First of all, modifiability to suit particular teacher's and student's needs is legally possible in the case of clear open license e.g. Creative Commons (clear and open license criterion) (MELT 2008; eQNet 2012). According to MELT (2008), modifiable LO should also be not text-heavy and easily translatable (language independence criterion). Besides that, it is also easier to modify LO having well-structured information, clear navigation and user-friendly interface (design and usability criterion) (Leacock, Nesbit 2007). LO modifiability is easier when its architecture is layered, i.e. its data, presentation and application logics are separated (architecture criterion) (Paulsson, Naeve 2006a). At last, according to MELT (2008), modifiable LO should be interoperable at least in terms of its metadata accuracy, modularity and adaptability, e.g. LO can be modified from a configuration file or from a plain text file and it should be suitable for localisation (interoperability criterion).


This LOs quality and reusability model was practically verified and validated during eQNet project by selected experts of 9 European Ministries of Education. Verification and validation confirmed that the model meets the needs of its users.

3.2. Example of evaluating quality and reusability of learning objects in eQNet project

A number of probably qualitative reusable Mathematics LOs have been identified while implementing eQNet project (eQNet 2012) and evaluated against the proposed MCEQLS TFN method.

Those LOs alternatives are as follows:

--L[O.sub.1]: "Mixed Numbers" (available from Internet: animations/mathematics/tilblanda/index.html);

--L[O.sub.2]: "Practice with Tangents and Circles" (available from Internet: http://regentsprep. org/Regents/math/geometry/GP14/PracCircleTangents.htm); and

--L[O.sub.3]: "How to Construct a Tangent to a Circle" (available from Internet: http://www.

The weight [a.sub.i] of the evaluation criterion reflects the expert's opinion on the criterion's importance level in comparison with the other criteria.

If the quality criteria are equally important for the experts-evaluators (let us call this "general case"), they should use the same weights for all the quality criteria according to normalisation requirement (4). If the experts-evaluators consider all criteria "very valuable" (i.e. TFN = 0.800 (Table 2)), we'll have the following matrix (Eq. (5)) of "general" (g) weights of the LOs quality criteria:

[a.sub.i] = [m.sup.i.sub.f]/[m.summation over (s=1)][m.sup.s.sub.f] = 0.800/8.000 = 0.100;

[a.sub.g] =(0.100 0.100 0.100 0.100 0.100 0.100 0.100 0.100 0.100 0.100). (5)

In real-life situations (e.g. in the case of analysing reusable LOs), different criteria are of different importance. In this case, the experts-evaluators should use different weights for different criteria. There could be different expert groups who determine the relative weights of criteria, and they could represent different education stakeholders, but all the experts should be qualified enough to evaluate the level of importance of both technological and educational LOs quality characteristics. If we pay special attention to LOs reusability level, we should establish higher weights to the 1st, 5th, 7th and 9th quality criteria (Table 1), because those criteria deal with LO reusability mostly. Therefore, according to formula (3) and Table 2, if we should establish "extremely valuable / essential" weights to these reusability criteria, and leave "very valuable" weights for the other quality criteria in Table 1, we'll get the following "reusable" weights:

[m.sup.1,5,7,9.sub.f] = 1.000 and [m.sup.2,3,4,6,8,10.sub.f] = 0.800.

After normalisation of these weights according to equation (4), we'll get the following matrix (Eq. (6)) of "reusable" (r) weights of the LOs quality criteria:

[a.sub.1,5,7,9] =[m.sup.1,5,7,9.sub.f]/[m.summation over (s=1)][m.sup.s.sub.f] = 1.000/8.800 = 0.114 and [a.sub.2,3,4,6,8,10] = [m.sup.2,3,4,6,8,10.sub.f]/[m.summation over (s=1)][m.sup.s.sub.f] = 0.800/8.800 = 0.091;

[a.sub.r] =(0.114 0.091 0.091 0.091 0.114 0.091 0.114 0.091 0.114 0.091). (6)

Now we can apply the experts' additive utility formula (Eq. (1)).

Criteria ratings (values) obtained while evaluating L[O.sub.1], L[O.sub.2] and L[O.sub.3] by the experts using TFNs are presented in matrix as follows (Eq. (7)):


The results of experimental evaluation of the analysed Math LOs "general" quality (g) using weights (Eq. (5)) and ratings / values (Eq. (7)) are presented in matrix as follows (Eq. (8)):

[a.sub.g] x f ([X.sub.j]) = (0.8200 0.7700 0.8200). (8)

The results of experimental evaluation of the analysed Math LOs "reusable" quality (r) using weights (Eq. (6)) and ratings / values (Eq. (7)) are presented in matrix as follows (Eq. (9)):

[a.sub.r] x f([X.sub.j]) = (0.8260 0.7810 0.8360). (9)

The obtained evaluation results mean that L[O.sub.1] meets 82.00% "general quality" (Eq. (8)) in comparison with the ideal, L[O.sub.2] - 77.00%, and L[O.sub.3] - 82.00%. They also mean that LO1 meets 82.60% "reusable" quality (Eq. (9)) in comparison with the ideal, L[O.sub.2] - 78.10%, and L[O.sub.3] - 83.60%.

Therefore, both L[O.sub.1] and L[O.sub.3] are the best alternatives (among the evaluated ones) from "general" quality point of view, but L[O.sub.3] is the best from "reusable" quality point of view.

4. Discussion

The method for evaluating quality and reusability of LOs proposed in this paper is simple to use and efficient. It is simple to use because it is based on simple scalarisation i.e. using the experts' additive utility function represented by equation (1), and no additional complex optimisation is required in this case. It is efficient (i.e. performing and functioning in the best possible manner with the least waste of time and effort) in comparison with the other analysed methods because it was widely and successfully applied during the last 3 years by 9 European Ministries of Education in eQNet project. Over 200 thousand LOs in European Learning Resource Exchange (LRE 2012) system for schools were analysed by eQNet experts, and over 5 thousand reusable LOs were identified there according to the MCEQLS TFN method and tagged "travel well" in LRE.

The authors have analysed the interactive and interrelated relationships among the criteria proposed by Antucheviciene et al. (2006) and have tried to adopt some other more comprehensive analytical tools (e.g. pair-wise comparison and other optimisation methods presented in Dzemyda et al. (2007) for possible use in the current study. Besides that, TFN method used in the current study to converse linguistic variables into non-fuzzy values to establish the values of the quality criteria and the weights of the quality criteria can be replaced by some comprehensive ways such as the one proposed by (Opricovic, Tzeng 2003) and (Wu, Lee 2007). However, the authors came to a conclusion that the proposed MCEQLS TFN method is the better alternative for the aims of the study in comparison with the other analysed methods because of its simplicity and efficiency. Besides that, eQNet experts also consider that the method proposed is effective, i.e. adequate to accomplish a purpose and producing the intended or expected result. Application of MCEQLS TNF method in eQNet have shown the similar results in comparison with evaluating LOs from the users' perspective in terms of number of LOs downloads, users' comments and star-ratings in LRE.

This method was also proposed to use to the other European Ministries of Education and publishers while evaluating quality and reusability of LOs. This will have a strong impact on further purchasing of LOs in the market by the Ministries of Education that have created EUN association, and on LOs creation process used by publishers.


Research results presented in the paper show that the original MCEQLS TFN method for evaluating quality and reusability of LOs (a) is applicable in real life situations when educational institutions decide on reusing LOs for their educational needs, and (b) could significantly improve the quality of the expert evaluation of LOs by noticeably reducing the expert evaluation subjectivity level. Experimental evaluation results show that the method proposed is quite objective, exact, simple to use, and efficient for selecting qualitative reusable LOs alternatives.

On the other hand, the proposed LO quality and reusability evaluation approaches are applicable for the aims of eQNet project in order to select LOs suitable to apply in different education systems in different European countries. Therefore, these approaches have been recommended by the authors to be widely used by European policy makers, researchers, publishers, practitioners (teachers), and experts-evaluators both inside and outside eQNet project.

Overall, MCEQLS TFN method is convenient in the case when there are many probably qualitative alternatives in the market, and it shows not only the difference between the quality of the evaluated alternatives expressed in per cent, but also the difference between the quality of the alternatives and the ideal quality.

There are some limitations of the presented research: since the authors' task was to propose simple to use and effective model and method for the expert evaluation of LOs quality and reusability, a very limited number of existing evaluation methods was deeply analysed. The large scale analysis and validation of suitability of both quality model and evaluation method proposed were performed in a large group of different education stakeholders, i.e. experts (researchers), Ministries of Education, publishers, and practitioners (teachers).

Future research work in the area could include an analysis of more comprehensive evaluation methods. LOs quality and reusability model could also be improved by individualising quality criteria according to different learners' needs, e.g. their prerequisite knowledge, preferred learning styles, etc.

Both the LOs quality and reusability model and method of TFN application to establish the quality criteria weights presented in this paper are absolutely novel and these new elements make the given work distinct from all the other/earlier works in the area.

Caption: Fig. 1. Trapezoidal fuzzy numbers

Caption: Fig. 2. Relationships between LOs reusability elements and quality criteria

doi: 10.3846/20294913.2013.837112

Received 22 May 2012; accepted 17 March 2013


Antucheviciene, J.; Turskis, Z.; Zavadskas, E. K. 2006. Modelling renewal of construction objects applying methods of the game theory, Technological and Economic Development of Economy 12(4): 263-268.

Balezentis, A.; Balezentis, T.; Brauers, W. K. M. 2012. MULTIMOORA-FG: a multi-objective decision making method for linguistic reasoning with an application to personnel selection, Informatica 23(2): 173-190.

Becta. 2007. Quality principles for digital learning resources [online], [cited 22 May 212]. Available from Internet: becta-schools-resources-digital-resources-quality-principles-digital-learning-resources-0

Belton, V; Stewart, T. J. 2002. Multiple criteria decision analysis: an integrated approach. Kluwer Academic Publishers.

Brauers, W. K. M.; Zavadskas, E. K. 2012. Robustness of MULTIMOORA: a method for multi-objective optimization, Informatica 23(1): 1-25.

Brereton, P.; Budgen, D.; Bennett, K.; Munro, M.; Layzell, P.; Macaulay, L. 1999. The future of software: defining the research agenda, Communications of the ACM 42 (12): 78-84.

Dzemyda, G.; Saltenis, V.; Tiesis, V 2007. Optimizavimo metodai [Optimisation methods]. Vilnius: Mokslo aidai, 144-159. (In Lithuanian).

eQNet (Quality Network for a European Learning Resource Exchange) project official site. 2012 [online], [cited 22 May 2012]. Available from Internet:

EUN (European Schoolnet) official site. 2012 [online], [cited 22 May 2012]. Available from Internet:

IEEE LTSC (the Learning Technology Standards Committee's of the Institute of Electrical and Electronics Engineers) official site. 2012 [online], [cited 22 May 2012]. Available from Internet:

ISO/IEC 14598-1:1999. Information technology--Software product evaluation - Part 1: General overview. 1st ed.

ISO/IEC 9126-1:2001(E). Software engineering--Product quality--Part 1: Quality model.

Koppi, T.; Lavitt, N. 2003. Institutional use of learning objects three years on: lessons learned and future directions. Presented at Edmedia 2001: World Conference on Educational Multimedia, Hypermedia & Telecommunications, 2003, Honolulu, Hawaii, USA.

Kurilovas, E.; Vinogradova, I.; Serikoviene, S. 2011. Application of multiple criteria decision analysis and optimisation methods in evaluation of quality of learning objects, International Journal of Online Pedagogy and Course Design 1(4): 62-76.

Kurilovas, E.; Dagiene, V. 2010. Evaluation of quality of the learning software. Basics, concepts, methods: Monograph. Saarbrucken, Germany: LAP LAMBERT Academic Publishing.

Leacock, T. L.; Nesbit, J. C. 2007. A framework for evaluating the quality of multimedia learning resources, Educational Technology & Society 10(2): 44-59.

Liu, P. 2012. An approach to group decision making based on 2-dimension uncertain linguistic information, Technological and Economic Development of Economy 18(3): 424-437.

LRE (European Learning Resource Exchange service for schools) official site. 2012 [online], [cited 22 May 2012]. Available from Internet:

McGreal, R. 2004. Learning objects: a practical definition, International Journal of Instructional Technology and Distance Learning 1: 21-32.

MELT (Metadata Ecology for Learning and Teaching) project official site. 2008 [online], [cited 22 May 2012]. Available from Internet:

Oliver, M. 2000. An introduction to the evaluation of learning technology, Educational Technology & Society 3(4): 20-30.

Opricovic, S.; Tzeng, G. H. 2003. Defuzzification within a multicriteria decision model, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 11(5): 635-652.

Ounaies, H. Z.; Jamoussi, Y.; Ben Ghezala, H. H. 2009. Evaluation framework based on fuzzy measured method in adaptive learning system, Themes in Science and Technology Education 1(1): 49-58.

Paulsson, F.; Naeve, A. 2006a. Establishing technical quality criteria for learning objects [online], [cited 22 May 2012]. Available from Internet:

Paulsson, F.; Naeve, A. 2006b. Virtual workspace environment (VWE): a taxonomy and service oriented architecture framework for modularized virtual learning environments--applying the learning object concept to the VLE, International Journal on E-Learning 5(1): 45-57.

Petkus, T.; Filatovas, E.; Kurasova, O. 2009. Investigation of human factors while solving multiple criteria optimization problems in computer network, Technological and Economic Development of Economy 15(3): 464-479.

Reigeluth, C. M.; Nelson, L. M. 1997. A new paradigm of ISD?, in Branch, R. C.; Minor, B. B. (Eds.). Educational media and technology yearbook 22: 24-35. Englewood, CO: Libraries Unlimited.

Song, W. W.; Andersson, M. B.; Hakkarainen, S. E.; Karlberg, P.; Soltesz, T. 1999. Metadata for the management of electronic documents in the governmental organisations and learning objects in the learning domain. SITI, SISU, Kista, Research report SITI 99:03, June, 1999.

Sosteric, M.; Hesemeier, S. 2002. When is a learning object not an object: a first step towards a theory of learning objects, International Review of Research in Open and Distance Learning 3.

Stanujkic, D.; Magdalinovic, N.; Stojanovic, S.; Jovanovic, R. 2012. Extension of ratio system part of MOORA method for solving decision-making problems with interval data, Informatica 23(1): 141-154.

Turskis, Z. 2008. Multi-attribute contractors ranking method by applying ordering of feasible alternatives of solutions in terms of preferability technique, Technological and Economic Development of Economy 14(2): 224-239.

Uppuluri, V. R. R. 1989. Prioritization techniques based on stochastic paired comparisons, in Karpak, B.; Zionts, S. (Eds.). Multiple criteria decision making and risk analysis using microcomputers, NATO ASI Series, F56, 293-303.

Vargo, J.; Nesbit, J. C.; Belfer, K.; Archambault, A. 2003. Learning object evaluation: computer mediated collaboration and inter-rater reliability, International Journal of Computers and Applications 25(3): 198-205.

Wiley, D. A. 2000. Connecting learning objects to instructional design theory: a definition, a metaphor, and a taxonomy [online], [cited 22 May 2012]. Available from Internet:

Wiley, D. A. 2002. The instructional use of learning objects, in Wiley, D. A. (Ed.). 1st ed. Bloominton: Agency for Instructional Technology and Association for Educational Communications & Technology, 15-19.

Wu, W. W.; Lee, Y. T. 2007. Developing global manager's competencies using the fuzzy DEMATEL method, Expert Systems with Applications 32(2): 499-507.

Zavadskas, E. K.; Turskis, Z. 2008. A new logarithmic normalization method in games theory, Informatica 19(2): 303-314.

Zavadskas, E. K.; Turskis, Z. 2010. A new additive ratio assessment (ARAS) method in multicriteria decision-making, Technological and Economic Development of Economy 16(2): 159-172.

Zavadskas, E. K.; Zakarevicius, A.; Antucheviciene, J. 2006. Evaluation of ranking accuracy in multicriteria decisions, Informatica 17(4): 601-618.

Zolfani, S. H.; Chen, I.-S.; Rezaeiniya, N.; Tamosaitiene, J. 2012. A hybrid MCDM model encompassing AHP and COPRAS-G methods for selecting company supplier in Iran, Technological and Economic Development of Economy 18(3): 529-543.

Zilinskas, A. 2005. Matematinis programavimas [Mathematical Programming] [online], [cited 22 May 2012]. Available from Internet: (in Lithuanian).

Eugenijus KURILOVAS (a), Silvija SERIKOVIENE (b)

(a) Vilnius Gediminas Technical University, Vilnius, Lithuania

(a) Institute of Mathematics and Informatics, Vilnius University, Vilnius, Lithuania

(b) Panevezys Institute, Kaunas University of Technology, Panevezys, Lithuania

Corresponding author Eugenijus Kurilovas


Eugenijus KURILOVAS is an Associate Professor in Vilnius Gediminas Technical University and Research Scientist in Vilnius University Institute of Mathematics and Informatics. He is a member of over 20 committees of international scientific journals and conferences, published over 80 scientific papers and 6 books, and participated in about 30 EU-funded large scale R&D projects and studies. He is also guest editor in "Journal of Universal Computer Science" and "Computers in Human Behavior", and reviewer in "Behaviour & Information Technology", "Computers in Human Behavior", "Journal of Web Engineering", and "IET Software", all abstracted / indexed in Thomson ISI Web of Science. He is the author of over 10 best paper awards in the largest international e-Learning conferences during last years.

Silvija SERIKOVIENE is a Lecturer in Panevezys Institute of Kaunas University of Technology. She has published a number of scientific papers in international journals and participated in a number of large scale R&D projects.
Table 1. Conversion of linguistic variables and into non-fuzzy values
to establish the values of the quality criteria

Linguistic variables   Trapezoidal non-fuzzy values

Excellent                          1.000
Good                               0.800
Fair                               0.500
Poor                               0.200
Bad                                0.000

Table 2. Conversion of linguistic variables and into non-fuzzy values
to establish the weights of the quality criteria

Linguistic variables             Trapezoidal non-fuzzy values

Extremely valuable / Essential               1.000
Very valuable                                0.800
Valuable                                     0.500
Marginally valuable                          0.200
Not valuable                                 0.000

Table 3. Learning objects quality model (criteria system)

Criteria     No   Quality criteria                 References
                  Technological quality criteria

"Internal"   1    Technological interoperability   Paulsson and Naeve
  quality         and reusability                    (2006a); Vargo
                                                     et al. (2003);
                                                     Leacock and
                                                     Nesbit (2007)
             2    Layered architecture             Paulsson and Naeve
             3    Technical stability and          Becta (2007)
Quality      4    Design and usability:            Vargo et al.
  "in use"          aesthetics, navigation,          (2003); MELT
                    user-friendly interface and      (2008); Leacock
                    information structure,           and Nesbit
                    personalisation                -2007

                  Pedagogical quality criteria

             5    Trans-national or                eQNet (2012)
                    cross-curricular topic
             6    Interactivity, strong visual     MELT (2008);
                    element                          eQNet (2012)
             7    Language independence            MELT (2008);
                                                     eQNet (2012)
             8    Ease of use, intuitiveness       MELT (2008);
                                                     eQNet (2012)
             9    Additional methodological        eQNet (2012)
                    support for teachers is not

                  IPR criterion

             10   Clear license: open, free to     MELT (2008);
                    use, cost-effective              eQNet (2012)
COPYRIGHT 2013 Vilnius Gediminas Technical University
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2013 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Kurilovas, Eugenijus; Serikoviene, Silvija
Publication:Technological and Economic Development of Economy
Date:Dec 1, 2013
Previous Article:Factors affecting customer loyalty of mobile RFID services in Korea.
Next Article:Organizational governance to integrate sustainability projects: a case study.

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters