A framework for the evaluation of visual languages for instructional design: the case of [E.sup.2]ML.

In the last years, Instructional Design has seen the development of visual notation systems for supporting and enhancing the design process. Some of them concern the definition of learning goals, while others address the definition of learning activities or learning materials. These tools supposedly reduce the cognitive load and enhance design communication. Few contributions try to assess the impact that such models have on the practice of Instructional Design. This article introduces a general framework for evaluation, indicating key issues and providing guidelines for the design of an evaluation program. As an example, an implementation of the framework along with the data collected about [E.sup.2]ML is presented.

Introduction: Instructional Design and Visual Languages

The design of instruction is a peculiar type of design, as its outcome is like the script of a play: it is an important element of the show, but not the only one. Your enjoyment of a theatre event also depends on the skills of the actors, on the performance of the orchestra, and on several other contextual elements such as the functioning of the heating system or the silence of the audience. Transposed to education, the quality of instruction does not only depend on the lesson plan and on the learning materials, but also on the ability of the instructor, on the mood of the students, etc. As Morrison, Kemp & Ross (2003, p.2) put it, "learning is haphazard; instruction is planned."

Design in complex situations requires conceptual tools for organizing the work both mentally and physically, and that is what happens with the aid of visual notation systems in Architecture, Mechanical Design, and recently in Software Engineering with UML (UML, 2001) or in Hypermedia Design with, e.g., W2000 (Garzotto, Paolini, Bolchini & Valenti, 1999). Visuals indeed allow a synthetic representation of complex objects and reduce the cognitive load (Blackwell, 1997; Lewalter, 2003).

Instructional Design (ID) models have always been visually supported (cf. the use of visuals in Dick, & Carey, 1996; Morrison Kemp & Ross, 2003; Greer, 1992). The difference with other disciplines is that such models represent the design process as a sequence of steps or a set of elements, and not the object being designed, as it happens in architectural blueprints. We could label these models instructional design process models--they will not be a topic in this article.

Instructional Design Languages

In more recent years some authors have claimed that, as other design-oriented disciplines, ID would benefit from a visual notation system that represents the instruction being designed (cf., Waters & Gibbons, 2004). Actually, the ID research tradition offers a very limited number of contribution in this direction. A part of them concerns the visualization of learning goals. To this class belong, for example, Merrill's Content-Performance Matrix (1983), the revised Bloom's taxonomy (Anderson & Krathwohl, 2001), and the Quail model (Botturi, 2004). The goal of such models is to provide a mental classification framework for learning goals, useful for discussing them and creating a common understanding within the design team.

Some novel works (Botturi, 2003; Belfer & Botturi, 2003; Belfer & Botturi, 2004) focused on the development of a blueprint language called [E.sup.2]ML--Educational Environment Modeling Language for representing the educational activity as such, thus producing a documentation of the design process. A similar approach was followed by other European researchers, who developed eduWeaver (Lischka & Karagiannis, 2004; Bajnai & Lischka, 2004) and the Person-Centered e-Learning patterns (Derntl & Mangler, 2004; Derntl & Motschnig-Pitrik, 2004). Finally, CADMOS-D (Psaromiligkos & Retalis, 2002; Retalis, Papasalouros & Skordalakis, 2002) is another language specifically developed for the design of web-based educational software applications.

Another interesting project under this respect is LAMS--Learning Activity Management System (LAMS, 2005). According to its authors, "LAMS is a revolutionary new tool for designing, managing and delivering online collaborative learning activities. It provides teachers with a highly intuitive visual authoring environment for creating sequences of learning activities." Its novelty is in fact the coupling of a learning management system with a visual design tool based on the IMS Learning Design standard (IMS, 2005). The beta testing phase, currently in progress, will reveal much of the potential of visual tools for ID.

Finally, another indication that the time is mature for a leap forward in the field of visual tools for instructional design is Reload (RELOAD, 2005), a recently released tool that provides a visual interface for creating IMS Learning Design instructional units.

In order to distinguish them from design process models I will call these contributions instructional design languages--understanding that they support the representation of the object being designed (e.g., learning goals, instructional activities or learning materials).

Goals and Structure of the Article

Despite the call of several authors for a continuous evaluation of design practices and tools (e.g., Osguthorpe & Zhou, 1989), few if any scientific contributions try to assess the impact of instructional design languages in the actual design practice. No validated answer is available to questions as: Does this model enhance the quality of instruction? Does it make the design process more efficient? Does it allow the implementation of more challenging solutions? This is probably due to the intrinsic complexity of ID, and to the complex relationships among the elements analyzed below.

The main goal of this article is therefore to propose a conceptual framework for supporting the development of evaluation studies about visual instructional design languages. In order to achieve this goal, the next section introduces some basic insights from communication theory and design theory, which will be reprised later, in the third section, for creating the evaluation framework. After the conceptual presentation of the evaluation framework, the article reports its first implementation for the set-up of an evaluation study of [E.sup.2]ML. The fourth section is therefore devoted to a short introduction of the language, and then reports the evaluation method and presents and discusses the data. The conclusions, concerning both the evaluation framework and [E.sup.2]ML, are presented in the last part of the article.

Notice that the article will not cover in detail any of the aforementioned languages, nor try to compare and evaluate them against each other (1). The development of a general evaluation framework is proposed here as a step toward a more widespread use of ID languages. In the first place, evaluation can promote a comparison of different languages; it can also provide evidence for a sensible choice of the language to be deployed in a specific design context.

Insights from Communication and Design Theory

Evaluating the impact of a design language means evaluating the impact of a new medium within a communication system. Medium is a word with several meanings, two of which are related to this case. In its most straightforward meaning related to communication, it can be defined as "a means or instrumentality for storing or communicating information" (WORDNET, 2005), such as the phone, or a CD. A language surely shares this nature. Yet a language is a medium for communication also as "the surrounding environment" (WORDNET, 2005), as in the sentence "fish require an aqueous medium"--communication requires a language in order to happen. Language is therefore a medium both as a tool that we use for communication, and as an environment in which communication events can exist and acquire meaning.

Language makes communication possible, or more affordable (Clark & Brennan, 1991), by creating a favorable environment. As fish have acquired a hydrodynamic shape through the evolution, so our communication will acquire a shape that makes it the most effective in our communication environment.

A language provides not only the words, but also the conceptual categories and metaphors through which we can express our experience and collaborate with others (Lakoff & Johnson, 1980). The introduction of a new language brings therefore potentially large effects. Given the complexity of communication processes, these are difficult to foresee. What is certain, is that these affects will not be reversible: a communication system is an ecological system, in which any new element changes the whole environment and sets a new balance (Cantoni & Di Blas, 2002).

The underlying implication is that design is a human activity which is tightly bound to communication. Design requires in fact a thorough shared understanding among all the parties that take part in the process: the stakeholders, the designers, and the developers. In a familiar setting, this means that the fashion designer should understand the self-perception of a woman in order to be able to prepare a wedding gown that suits her and that she likes; he must then be able to communicate his design to the tailors who will realize it and to guide their activities. This happens with music, dance, mechanical engineering, architecture, etc., and of course, ID: communication plays a major role in any kind of design (Waters & Gibbons, 2004).

The development of an evaluation framework for instructional design languages should rely on a sound understanding of the communicative processes involved. In order to provide some background for the proposal presented in this article, the following paragraphs present four insights from communication and design theory. I will start from the idea of language as a medium that extends communication possibilities, and I will then move on to discuss the cognitive and social impact of a new language through the concepts of metaphor and of "massage." The last insight concerns more strictly the relationship between conceptual language and notation system in design.

Languages and Technologies of the Word

Walter Ong (1982) presents writing and then the evolution of (mass) media as the "technologizing of the word," for instance, as the process of creation of tools that amplify the human ability for verbal communication. The author claims that this evolution is not without consequences: a change in the structure of media affects the people who use them, their perception of the world and of their experience. This is clear when one observes the difference between writing-oriented cultures (as the Western modern culture) and orality-based cultures. For example, the members of a culture that extensively uses writing will not be concerned with the development of memory and of forms of personal transmission of knowledge, but can rely on artifacts--exactly as we rely on documents, books and the Internet for storing and retrieving information; information can be objectified into artifacts. Oral cultures do not share this perspective; information is intrinsically connected to people, is volatile, and has therefore a different relationship with individuals and groups, which leads to different forms of social structure.

Under this respect, a design language can be interpreted as a specific technology of the word, as it offers an extension of the communication possibilities in a specific context. It allows one to write on paper things that were before only told, and to create new communicative situations. This idea of language as extensor of communication possibilities will be reprised below, when addressing the topic of institutional changes and of impact on communication activities.

Languages and Metaphors

Lakoff & Johnson (1980) claim that our perception of the world is based on metaphors. As humans, we do not know the nature of reality, and we try to make sense of it through metaphors that relate to our experience. So we say that "it's a hard life," that some guy "is cool," or that "I'm down," or even "visual languages are a new topic in the field."

The source of these metaphors is our language. A language is therefore to consider not only as a communication vector, but as the source of the concepts and basic metaphors through which we understand the world. A source that was created and maintained with the tradition of a people of speakers (2). It is clear that a new language opens up a new landscape for the community of speakers.

Under this respect a design language is a peculiar type of language, shared by a small community and focused on a specific task. Inns (2002) and Schon (1993) describe metaphors as tools for generative thinking: a new language in a (design) community can bring to the development of new ideas, as it offers metaphors that allows a new approach to old issues. On the other hand, a new language also brings new limitations not only in communication, but also in thinking, as it can inhibit the use of specific metaphors. What the effects could be, is not completely foreseeable, as it depends on the reactions and creativity of the speakers.

These insights will form the main background in the introduction of the concepts of expressive power (what metaphors allow to express), eclectic benefits (what metaphors allow to do) and context sensitivity (how metaphors integrate with other languages and practices).

The Medium is the Massage

In an incredibly modern booklet published way back in 1967, McLuhan & Fiore play a pun on words and describe the effects of a medium as a massage [sic]. Their point is that a new medium, as a new communication tool or a new language, does not bring a sudden revolution, rather smoothly presses on our perception and experience and slowly brings forth huge modifications.

A language creates--step by step--a new communication environment, where new concepts are used (see the idea of metaphors, above) and new expressions are possible, while some old concepts and expressions might become out of date, or even not possible any more. It is easy to see this development comparing e.g., TV ads from the past five decades and focusing on the visual language they use, their messages, and their rhythm. And in fact, no TV spot designed in the ways it would have been designed in the '40s would be effective today. This remark suggests to consider with a particular care the issue of time: the changes that a medium brings forth do not happen all at once, but require time, and should be observed and understood in-depth.

Design Languages and Notation Systems

All the efforts mentioned in the Introduction aim at the development of both a basic conceptual language and a user-friendly visual notation system. In order to gain a better understanding of the problem, a definition of these two terms is required. We propose here the distinction proposed by Waters & Gibbons (2004). According to their definitions, a design language is a personal and abstract set of concepts that a designer can use for creating design structures. It corresponds to the "source of metaphors" that was discussed above.

A notation system is a tool for providing imperfect but visible and public expression to design structures. This introduces a new element in the discussion, as up to now I just discussed about languages in general, without distinguishing between the idea of language and how languages are used in practice. In other words, an architect creates a new and original building thanks to a set of aesthetical, compositional and technical concepts, and then expresses her ideas through a set of drawings that allow her to share the project with other people.

As Waters and Gibbons emphasize, there is a tight relationship between design languages and notation systems, as between our thought and our mother tongue. "As designers improve and extend their personal design languages, this in turn calls for extensions and improvements to the notation system. The notation system then is capable of expressing more interesting and complex designs and easily leads to innovation." (Waters & Gibbons, 2004; p. 59). This looping relationship is summarized in Figure 1.

The cycles through which a notation system suggests new terms--or new metaphors--for the design language, might require time and might occur slowly, as discussed above about the "massage." These considerations will play a major role in the evaluation framework for the introduction of the concepts of time and expressive power.

[FIGURE 1 OMITTED]

A Framework for Evaluation

The evaluation of a language--although a design one--is not an easy task. The use of a language is the result of complex interactions among the speakers and among the community of speakers and other communities, and its effectiveness is tightly connected to creativity. In some sense, a language is a flexible and continuously developing tool, evolving into dialects and jargons under the pressure of the situations the speakers have to get through.

Moreover, the specific domain of ID is manifold, as each organization and design team has its own practices (Schwier, Campbell & Kenny, 2003). At the current stage every designer, or small group of designers, has her/his own idiosyncratic language that fits specific needs (Waters & Gibbons, 2004).

The main idea of having an evaluation framework is to provide a limited set of well-defined concepts that can guide the set-up of evaluation studies in this area. Such a framework (see Figure 2) is required if we consider, as already mentioned, that there are to date no studies of this kind, although a number of design languages are currently being developed.

The framework proposed in this article is built upon two groups of items: issues and elements.

1. Issues are critical aspects that should be considered in the definition of the experimental setting. They are: context sensitivity, eclectic benefits, course quality, and time.

2. Elements are indications for the identification of key variables in the study. They are: impact on sub-activities, impact on communication events, institutional changes, and expressive power.

The following paragraphs present the framework in detail. The next section reports a first implementation of the framework for [E.sup.2]ML. Its goal is to see how the items in the framework actually guide the set-up of an evaluation study and to discuss the kind of results that are observed.

Evaluation Issues

The evaluation of an instructional design language has to cope with a number of issues, some related to design in general, others specific of the educational setting. The following paragraphs introduce four key issues, specifying for each of them indications to be taken into account in the definition of the experimental setting.

Context Sensitivity. The actual use and effectiveness of a new design language strictly depend on the impact that its metaphors have on the metaphors currently in use. This means that a central role is played by the designer, the type of instruction to be designed, and the overall institutional and educational context. The complex connection between these elements makes it difficult to define a standard evaluation protocol. For example, [E.sup.2]ML is suitable for system-level design; nevertheless, while some courses would benefit from it (for example, a mixed-mode course), other courses even in the same institution may not (for example, a face-to-face lecture series). At the same time, some designers may feel so familiar with it to use it also for quick design of small courses, where it would otherwise not be useful. The Person-Centered Design Patterns, for example, are based on UML--any community of designers working with software engineering subject matter experts should consider it as a straightforward communication device with their partners.

[FIGURE 2 OMITTED]

An evaluation study should therefore carefully describe the context of use, the size and composition of the community of speakers/users, and the design process in which the language is used. It is important to clearly specify the organizational and operational context of design (department, e-learning development team, single teacher, etc.), the types of instruction being designed (course, lecture, instructional unit, etc.), the competencies and background of the designers, and the goals and constraints in using the language (imposed by the administration, experiment, free personal choice, etc.).

Eclectic Benefits. In order to evaluate the impact of a tool, one should determine what the expected benefits are for its users. A language may bring a number of different benefits, but only some of them could be achieved in a single instance situation. For example, some may use a visual language as it makes course revisions easier, although it requires some additional time for the first design; others may use it as a standard visualization for all courses, so that any designer can quickly get the rationale of any course, etc. The (ben)efits depend on the metaphors that the language brings; and how they are integrated with already existing languages and practices.

It is therefore necessary for evaluation studies to declare what the expected benefits are--both from the point of view of the evaluators and from that of the designers. Such expectations will determine in what situations and to what extent the language will be used, and, of course, what language to choose among those available.

Course Quality Assessment. A relevant element in the evaluation of a design language is the quality of the product, as a result of the communication process. Yet the uniqueness of each educational environment, as a whole composed by a subject matter, a method of instruction, a class, the teaching staff and the learning materials, makes quality assessment of a single course problematic, as the large number of pages about evaluation in the literature testifies, especially about ROI (Kirkpatrick, 1998). Formative and summative evaluations in fact are measures of the intrinsic quality of a course and of its adequacy to the goals for which it has been developed, and could not be used as comparative values without a strong bias. Is a course a good course because all learners achieve its objectives, although none of them were able to do any other course in the same term because of work overload? Is a course a good course because the 3D animations developed for it won a prize, although the course overspent budget? The elements to be considered are many--strictly pedagogical, administrative, institutional, etc.--and are often tightly intertwined. The comparison of two courses increases the difficulty, as no parameters can be set for both of them in order to identify variables. For example, no two courses on the same topic may have the same class with the same entry level; and no two courses can be designed by the same designer with the same level of expertise.

It is therefore necessary for the evaluation studies that want to consider course quality as a parameter, to split the different dimensions, and analyze them separately: learning quality, media development, management and budget, etc.

The Importance of Time. As mentioned above in relation to the idea of "massage" and to that of cycle between design language and notation system, time is of paramount importance for the integration of a language in a community's practice. The progressive rearrangement and smoothening of the language through use is a deciding process, as a community's language should be developed and negotiated by the community itself. It could be taken as hypothesis that the introduction of a language would follow the pattern of innovation diffusion (Rogers & Shoemaker, 1971). It is indeed likely that it would lower productivity for a little while, raising it afterward. A complete evaluation program should therefore observe the evolution of the design practice and of the quality of the instruction over a long period and over more courses. Smaller efforts, such as the one presented in this article, should clearly define and declare their observation time span, and balance the observed effects adequately.

The issues presented above are only a part of those that must be considered in the evaluation of an ID language; they were selected because they are often overlooked and entail potential danger. Others could be listed--no two design teams work the same way; the choice of technologies has an impact on design; the different personal and cultural degrees of openness and will to collaborate of designers and instructors matters, etc.

Evaluation Elements

Given a well-defined experimental setting, research should determine what indicators to observe. The following paragraphs try to put forth some hints in order to provide a focus for guiding the identification of relevant indicators.

Impact on Sub-activities. The quality of a tool is its adequacy to a problem-solving activity for its users (Hoyer & Brooke, 2001). Instructional design is a complex activity which involves different steps and sub-activities which are different in nature, such as analysis, development and evaluation. Given that a design language provides metaphors and conceptual tools for specific tasks, it is advisable to select limited sub-activities to observe for evaluation. An example would be a new designer in charge of redesigning two courses developed by someone else: she has only the course materials for the former, and a complete documentation e.g. in [E.sup.2]ML for the latter. Her evaluation of her own work, and of the aid of the documentation, along with a measure of effectiveness (e.g., time spent), would offer a measure of the impact of a design language on a particular situation. A sound evaluation of the use of a design language in a complete ID process, should split the data collection for the different activities (analysis, design, development, etc.).

Impact on Communication Events. Among the sub-activities of a design process, communication events should be observed with special attention. They are particularly relevant as a design language is a communication device--its effects are rooted in the collaborative nature of design, and they are the main indicator of change in the communication environment, as mentioned above. For example, the meetings of a design team could be videotaped in order to see the role that diagrams play when discussing objectives or activities. The effectiveness of meetings could be partially assessed measuring their duration and recording the judgment of the designers who took part in them. The use of diagrams for involving other stakeholders could also be another interesting point. Communication events should be therefore one of the main focus points in an evaluation program.

Institutional Changes. Both the contributions of McLuhan and Fiore (1967) and Ong (1982), led to the identification of social changes as one of the effects of new media and new languages. A design language actually provides the possibility to create a shared repository of courses, or to define pedagogical patterns, etc. Moreover, observation could include the training of novice designers, the sharing of expertise and best practices, the reuse of design, and the communication inside and outside the team as elements of knowledge management. In the context of an organization, the guidance of the integration would as well be at stake: who is sponsoring the introduction of the new language? What are the major drivers? What the perceived benefits and fears? A long-term study of instructional design languages should also focus on the social and organizational or institutional dimensions of change.

Expressive Power. One of the most important intrinsic features that make a language useful is its expressive power, for instance, the extension of the domain of objects that it can describe through the words and metaphors it makes available to the speakers. Can it equally well represent instruction delivered with different media, or in different settings? Can it grasp the essence of different pedagogical approaches? On this point, researchers should pay attention to the distinction between conceptual language and notation system mentioned above. Imagine you are evaluating a visual language for courses with a constructivist approach. It might happen that the language can be extended to include some new kind of activity conceived by the instructor, although the current visual notation system makes it difficult to express it. In this case the language semantic structure is flexible and expressive, but the notation system is not.

A focus on the expressive power would be the most sensible first step for a comparison of different ID languages.

Implementation Case: An evaluation of the Perception of [E.sup.2]ML

This section reports a preliminary evaluation of [E.sup.2]ML--Educational Environment Modeling Language. The study, conducted between May and September 2003, was designed in order to assess the first impression that experienced designers got from the language in terms of usefulness for their practice.

At first I will briefly introduce the language. For reasons of space and according to the goals of this article, the introduction will be far from complete: its goal is just to provide the necessary insight for understanding the evaluation process, while all necessary references for complete information about [E.sup.2]ML are provided in the text. I will then show how the general evaluation framework was implemented for this language, presenting the tools and results of the evaluation process.

All the examples are taken from a two-day course in Effective Mediated Communication (EMC) for commercial managers.

About [E.sup.2]ML

[E.sup.2]ML is a visual language for the design of educational environments. The main issue [E.sup.2]ML is concerned with corresponds to what Greer (1992) and Reigeluth (1983) called the development of a blueprint: a representation of the instruction that all stakeholders, designers, developers and instructors can see, understand in a similar way and, hopefully, agree upon. The development of an [E.sup.2]ML blueprint means modeling the instruction into a set of documents that provide a support for the people involved in the design process. The documentation is organized into three document sets.

Goal Definition. It is a declaration of the educational goals. This is composed by two documents: the goal statement and the goal mapping, which exploits a goal visualization model. An example of goal statement table is reported in Table 1, and its corresponding visual mapping in Figure 3. The visual mapping was developed using the Quail model (Botturi, 2004), although other schemas could be used as well. Roughly, the Quail model represents each learning goal as a dot with a different shape, corresponding to the type of knowledge addressed (fact, concept, procedure, etc.). Goals are then located onto a grid formed by knowledge levels (vertical axis) and scope (a sort of level of application corresponding to Merril's Content-Performance Matrix--see Merrill, 1983).

[FIGURE 3 OMITTED]

Action Diagrams. Action diagrams are a description of the single learning and support activities designed for the instruction. Each activity is represented as a table that contains short descriptors for each feature: who takes part in the activity, the prerequisites, the expected outcomes, etc. Action diagrams also bear a reference to goals.

Overview Diagrams. Finally, there are two overviews of the whole design. The dependencies diagram shows the reletionships among activities (Figure 4), for example, if an activity requires another one in order to be completed. The activity flow is a sort of visual calendar (Figure 5), similar to those produced with the LAMS application tool (LAMS, 2005).

[E.sup.2]ML aims at enhancing design through enhancing communication among those who do it. Its emphasis is on visualizing and sharing design ideas and solution with the design team and with external partners and stakeholders, rather than scaffolding strong design through rules. As any real design process and any real instructional situation has its own unique features, the language can (and should) be adapted, simplified or detailed, to the needs of the specific context or design team. The documents are produced at different moments in the design process, and do not have a tight correspondence with specific phases (3).

[FIGURE 4 OMITTED]

[FIGURE 5 OMITTED]

Experimental Setting

The evaluation reported in this article was structured according to the concerns and elements of the general framework presented above. It has a narrow focus, as it was developed to investigate a very specific population and moment in time, as explained below. Nevertheless, its findings provide (a) valuable inputs for improving [E.sup.2]ML; (b) indications for its use in the practice; and (c) evidence to compare it with other languages. The implementation of the framework is represented in Table 2.

The data collection took two main forms: (a) two focus groups were held at DE & T, University of British Columbia, Vancouver, involving 10 designers with different backgrounds (computer science, media production, instructional design, education, and religious studies) and different ways of doing design; (b) 12 designers from different institutions were individually interviewed and then required to fill in a feedback form after they had assisted to a group or individual presentation of [E.sup.2]ML.

Focus Groups

The focus groups were designed as an attempt to focus the participants on the specific activities and communication events that were selected for the study: definition of strategies, set up of learning activities, learning material development, and meetings with stakeholders and partners. The overall impression that all designers expressed is that [E.sup.2]ML looks potentially powerful, flexible and adaptable to different design strategies and situations. Its main innovative feature is its visual orientation, which provides a synthetic view of the instruction: they confirmed that they usually develop a mental image of the course that they never express, if not implicitly in the course materials, and that can be visualized with [E.sup.2]ML, providing an interesting focus for the discussion in the design team.

According to their perception, [E.sup.2]ML is mostly useful for keeping the overall consistency of a course, and in particular to discuss the consistency of goals and instructional activities with the instructors or course authors, who usually discuss the goals and then forget them in the actual planning, so that a consistent strategy is something difficult to explain, and visualization is an important support. Moreover, designers felt that it could be useful to blueprint a course, as it works well in organizing people's thinking, and may speed up collaboration, also allowing a greater detail than usual textual blue-prints or learning material drafts.

While designers feel [E.sup.2]ML could be learnt in a reasonable amount of time, its complexity may make it difficult for instructors and course authors--it has a steep learning curve. From this point of view, visual learners might be favored, although designers do not think this is a prerequisite for using [E.sup.2]ML. Given the usual time constraints--which seems to be unavoidable in the ID practice--and the burden of administrative work that goes with it, (Cox & Osguthorpe, 2003), designers feel they would like to use a new tool only if this does not take too much time to be mastered. Besides the design activity, [E.sup.2]ML is felt as a possible support for communicating the structure of activities to the students. From a practical point of view, all designers agree that [E.sup.2]ML should come with templates or a specific software application, which might also save some learning time. Finally, two more formal considerations emerged, concerning its expressive power: (a) the flexibility of [E.sup.2]ML with regard to learning objects on the one hand, and the necessity of a specific product-oriented model for the development of specific resources; and (b) time and durations of learning activities are not evident in [E.sup.2]ML.

Feedback Interviews

Interviews were a semi-structured discussion of the model based on some cases inspired by the implementation choices made with the framework, and confirmed the results of the focus groups, providing important elements for their correct interpretation. Interviewees were then asked to formalize their answers filling in a short feedback form. The feedback form was organized in two main parts:

1. Scenarios: designers were presented short descriptions of situations, and then asked if [E.sup.2]ML would have been a support for the specific instance. Answers were to be marked on a Likert-type scale.

2. Statements: designers were presented some general statements about [E.sup.2]ML, and they were asked to check the ones they felt true. Half of the statements indicated positive features, half negative ones.

Scenarios

The scenarios presented different typical situations in ID, and each of them was conceived as representative of a specific design activity (e.g., team organization) or instructional feature (e.g., consistency). Table 3 reports a sample of the scenarios descriptions, along with a synthetic statement of the activity/feature at stake.

The scenarios results are summarized in Figure 6: each feature is represented as a bar, as indicated in the chart key. Values go from 0 (the feature is not supported by [E.sup.2]ML) to 2 (the feature is well supported by [E.sup.2]ML). Intermediate values should be intended as degrees of possibility--1 means something like, "It is possible to use [E.sup.2]ML in order to do that, but it would require some rearrangement."

All interviewed designers basically felt that all the proposed features were supported by the language. In particular, all of them expressed confidence that it can enhance team communication (bar 1) and support the comparison of different designs (bar 2). Also very high confidence was expressed for the use of [E.sup.2]ML as a language for keeping the overall consistency of the instruction (bar 3), adapting a course when the instructor changes (bar 4) and for teaching novice designers (bar 5).

The use of [E.sup.2]ML for the adaptation of existing designs with different students (bar 6) has a slightly lower score. Designers feel that [E.sup.2]ML may be useful for working with the instructor, while changing student target often means redesigning the course form scratch. Comments about these scenarios pointed out that the rationale of a course is given by the epistemological beliefs of the instructor--Richards & Rodgers' (1982) approach layer--and that often effective learning depends more on that than on the design of specific activities or on the quality of support materials--the design layer--where [E.sup.2]ML seems to be more applicable.

The use of [E.sup.2]ML for checking the implementation status (bar 7) also got a middle confidence score, while lower confidence was expressed regarding the use of [E.sup.2]ML for the development of instructional materials (bar 8)--designers feel that it is too high-level for implementation, and that what they usually pass to Web programmers is a more specific description, or some content to be put into HTML pages. Noticeably, the lowest confidence is for two important elements: the identification of workload (bar 9) and the use of [E.sup.2]ML as a diagnostic tool, for example, to identify negative unexpected learning outcomes (bar 10). Although both of them got a final score above 1, the result shows a large space for improvement.

Statements

The statements indicated positive and negative features concerning the overall expected impact of the language. One last statement concerned the development of a software application for the creation of [E.sup.2]ML diagrams. They are the following:

* Positive statements:

- [E.sup.2]ML can enhance the quality of instruction.

- [E.sup.2]ML can support the implementation of more challenging design solutions for education.

- [E.sup.2]ML can make the design process smoother.

* Negative features:

- [E.sup.2]ML is too complicated.

- [E.sup.2]ML has too many elements.

- The effort [E.sup.2]ML required in writing the documentation is not rewarded anyway.

* [E.sup.2]ML would be nice if it could be used with a software application.

Unlike for scenarios, designers had here a binary choice: the statement applies or not. The results are reported in Figure 7, where the values (between 0 and 1) represent the percentage of people that checked each item.

At a first sight it is clear that positive features are felt more correspondent to the reality than negative ones. Remarkably, all designers think that [E.sup.2]ML can enhance the quality of the instruction, and a great part of them that it can smoothen the design process. Moreover, only few think that it is too complicated, and a very small part finally thinks that it has too many elements and that the effort eventually spent in learning and using [E.sup.2]ML might be too large with respect to the return. Finally, the development of an [E.sup.2]ML application would be welcomed by the largest part of the interviewed designers.

DISCUSSION

After a short state of the art summary, the first part of this article proposes a general framework for the evaluation of visual languages in ID, providing a set of guidelines and critical issues that should be taken into account. The framework is guide for the set-up of sound evaluation studies of the impact of instructional design languages, and is based on communication and design theory. The second part of the article provides an example of application of the framework to the evaluation of [E.sup.2]ML, a novel visual blue-print language. The data collected offer interesting insights about the use of visual languages in ID: designers expressed an overall positive impression about [E.sup.2]ML, which they considered an interesting new tool showing potential usefulness for their practice.

About the evaluation framework

The evaluation framework introduced in this article provides a sound guidance for the set-up of evaluation studies in this area. The implementation case presented has proved that it is usable and useful, and the results obtained actually provide interesting inputs both for the further development of [E.sup.2]ML and for its use. The framework provides guidelines for defining the elements for the evaluation, both the experimental setting and the indicators to be observed. In the implementation case, it has provided guidance to define the limitation of the considered time span, the focus on specific elements as expected benefits, the non-emphasis on course quality, the selection of a specific institutional context. It is hoped that the framework provides a structure by which evaluations of the impact of different languages are comparable, and that practitioners can find some support in selecting what language to use in their practice. Future implementations of the framework and new insights in communication and design theory will help in assessing its completeness and soundness.

About [E.sup.2]ML

Far from being a complete evaluation with a definitive claim--also given the small sample--the data presented above were collected with the goal of providing an initial measure of the first impression of [E.sup.2]ML on experienced designers. The evaluation results provide hints about the possible uses of [E.sup.2]ML. In fact hey show that [E.sup.2]ML is felt (although not yet experienced by designers) as a powerful communication tool, especially for comparing designs. Under this respect, it is useful for selecting alternatives or comparing courses. It is also interesting that the initial overhead spent for the creation of the documentation can be rewarded by the benefit the documentation brings to course revision processes. From a general point of view, the results indicate that visual instructional design languages are perceived as potentially powerful tools, provided that they are flexible, adaptable and easy to use. A natural follow-up for the evaluation of [E.sup.2]ML would be the assessment of its actual use and impact in the long term in a community of designers, considering also the institutional changes it fosters and the quality of courses, (i.e., by introducing different variables in the general framework).

As a general contribution to ID research, this article provides some initial evidence that designers see visual ID languages as interesting new possibilities, and provides indications about their features and the evaluation of their impact on the practice.

References

Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching and assessing. A revision of Bloom's taxonomy of educational objectives. New York: Addison Wesley Longman.

Bajnai, J., & Lischka, J. (2004). Planning and simulation in an e-learning engineering framework. EDMEDIA 2004, Lugano, Switzerland.

Belfer, K., & Botturi, L. (2003). Pedagogical patterns for online learning. ELEARN 2003, Phoenix, Arizona, USA.

Belfer, K., & Botturi, L. (2004). Online learning design with pedagogical patterns. SALT Orlando Conference 2004, Orlando, Florida, USA.

Blackwell, A.F. (1997). Correction: A picture is worth 84.1 words. In C. Kann (Ed.), Proceedings of the First ESP Student Workshop, 15-22.

Botturi, L. (2004). Visualizing learning goals with the quail model. Australasian Journal of Educational Technologies--AJET 20(2), 248-273.

Botturi, L. (2003a). [E.sup.2]ML--A modeling language for technology-dependent educational environments. EDMEDIA 2003, Honolulu, Hawaii, USA.

Botturi, L. (2003b). [E.sup.2]ML--Educational environment modeling language. Ph.D. Thesis. Retrieved on January, 14th 2005 from http://www.istituti.usilu.net/botturil/web/e2ml/index.htm.

Cantoni, L., & Di Blas, N. (2002). Teoria e pratiche della comunicazione. Milano, Italy: Apogeo.

Clark, H.H., & Brennan, S.E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition (pp. 127-149). Washington, DC: APA.

Cox, S., & Osguthorpe, R.T. (2003). How do instructional designers spend their time? Tech Trends 47(3), 45-47.

Derntl, M., & Mangler, J. (2004). Web services for blended learning patterns. IEEE International Conference on Advanced Learning Technologies, Joensuu, Finland.

Derntl, M., & Motschnig-Pitrik, R. (2004). Patterns for blended, person-centered learning: Strategy, concepts, experiences, and evaluation. ACM Symposium on Applied Computing (SAC04), Nicosia, Cyprus.

Dick, W., & Carey, L. (1996). The systematic design of instruction (4th edition). New York: Harper Collins College Publishers (6th edition published in 2001).

Garzotto, F., Paolini, P., Bolchini, D., & Valenti, S. (1999). Modeling-by-patterns of web applications. In International Workshop on the World Wide Web and Conceptual Modeling (WWW CM '99), Paris, France.

Greer, M. (1992). ID project management: Tools and techniques for instructional designers and developers. Englewood Cliffs, NJ: Educational Technology.

Hoyer, R.W., & Brooke B.Y. (2001). What is quality? In: Quality Progress, 34(7), 53-62.

IMS (2005). IMS learning design standard specification. Retrieved on January, 14th 2005 from http://www.imsglobal.org/learningdesign/index.html

Kirkpatrick, D. (1998). Evaluating training programs: the four levels (2nd ed.). San Francisco: Berret-Koehler.

LAMS (2005). Learning Activity Management System foundation website. Retrieved on January, 14th 2005 from www.lamsinternational.com.

Lakoff, G., & Johnson, M. (1980). Metaphors we live by. London: University of Chicago Press.

Lewalter, D. (2003). Cognitive strategies for learning from static and dynamic visuals. In Learning and Instruction, 13, 177-189.

Lischka, J. & Karagiannis, D. (2004). Modeling and execution of e-learning resources. ACM Symposium on Applied Computing (SAC04), Nicosia, Cyprus.

McLuhan, M., & Fiore, Q. (1967). The medium is the massage [sic]. New Work: Bantam Books.

Merrill, M.D. (1983). Component display theory. In C.M. Reigeluth, (ed.), Instructional-design theories and models: An overview of their current status. NJ: Lawrence Erlbaum Associates Publishers, 279-333

Morrison, G.R., Ross, S.M. & Kemp, J.E. (2003). Designing effective instruction (4th edition), NJ: Wiley & Sons.

Ong, W. J. (1982). Orality and literacy: The technologizing of the word. London: Methuen.

Osguthorpe, R.T., & Zhou, L. (1989). Instructional science: What is it and where did it come from? In: Educational Technology, 29(6), 7-17.

Psaromiligkos, Y. & Retalis, S. (2002). Configuration Management for Web-Based Instructional Systems. In 2nd International Workshop on Web Oriented Software Technology, Malaga, Spain

Reigeluth, C.M. (1983). Instructional-design Theories and Models: An Overview of Their Current Status. NJ: Lawrence Erlbaum Associates Publishers

RELOAD (2005). Reload Website. Retrived on March, 4th, 2005 from http://www.reload.ac.uk/

Retalis, S., Papasalouros, A., & Skordalakis M. (2002). Towards a generic conceptual design meta-model for web-based educational applications. In 2nd International Workshop on Web Oriented Software Technology, Malaga, Spain.

Richards, J., & Rodgers, T. (1982). Method: Approach, design, and procedure. Journal for Teachers of English to Speakers of Other Languages Quarterly, 16(2), 153-168.

Rogers E., & Shoemaker, E. (1971). Communication of innovations, A cross-cultural approach (2nd edition). New York: The Free Press.

Schwier, R.A., Campbell, K. & Kenny, R.F. (2003). Instructional designers as change agents: Perceptions of communities of practice. Paper presented at the Annual Conference of the Association of Media and Technology in Education in Canada--AMTEC 2003, Montreal, Canada.

UML (2001). UML 1.5 Specification. Retrieved on January, 14th 2005 from www.uml.org.

Waters, S., & Gibbons, A.S. (2004). Design languages, notation systems, and instructional technology: A case study. Educational Technology Research & Development, 52(2), 57-68

Notes

(1) Under this respect, given the novelty of the visual language trend in ID, the efforts are still to come and there is currently no evidence. The aim of comparing different languages is currently pursued by a growing network of researchers interested in the topic in Switzerland, Austria and the US.

(2) By the way, this is also the most difficult thing to learn in a foreign language: not words, grammar or syntax, but the metaphors that native speakers use. In Italian someone who's really talkative and boring is "heavy"--a metaphor that doesn't make a lot of sense in English. Also, "making sense", literally, means "is disgusting" in Italian, which uses a different metaphoric system.

(3) A complete introduction to the language is available in (Botturi, 2003a), (Botturi, 2003, b) and (Belfer & Botturi, 2004).

LUCA BOTTURI

University of Lugano, Switzerland

[e-mail]

Table 1 Goal Statement for EMC

Goal Statement
Tag  Statement             Target  Stakeholder  Approach      Importance

A1   Recognize critical    All     Head         Case          5
     success factors in                         studies and
     communication                              discussion
A2   Analyze successful    All     Head                       4
     and unsuccessful
     communications
A3   Recognize             All     Head                       5
     differences
     between direct and
     mediated
     communication
     settings
B1   Recall key concepts   All     Head         Critical      4
     of communication                           discussion
                                                on movie
                                                clips
C1   Perform effective     All     Head         Guidelines,   2
     videoconferencing                          examples
                                                and
                                                exercises
C2   Perform effective     Head                               3
     audioconferencing
     All

Table 2 Implementation of the Evaluation Framework

Issues
Context sensitivity      The institutional context was determined by
                         selecting the target population: feedback was
                         collected from practitioners employed as course
                         designers or course developers in ID
                         departments in universities on the West coast
                         of Canada and the United States
Eclectic benefits        The study focused on the perception of
                         usefulness in communication with stakeholders
                         and design management
Course quality           The evaluation was not based on the quality of
                         courses
Time                     The evaluation was focused on a specific moment
                         of the introduction of the language in the
                         design practice--the very first introduction

Elements
Specific sub-activities  Definition of strategies and the set-up of
                         learning activities, along with the development
                         of learning materials
Communication events     Expected impact on meetings and communications
                         with partners and stakeholders
Institutional changes    Not considered (limited time span)
Expressive power         Not considered (although some indications
                         emerged)

Table 3 Sample Scenarios

Scenario                                         Key

You are in the development team for a course in  Team communication
Economics along with a faculty, a subject        enhancement
expert from the corporate world and a Web
programmer. It looks like you talk different
languages and it is not easy to understand each
other. Would [E.sup.2]ML enhance internal team
communication?
You are tight on schedule with a course, and     Material development
you run to the Web programmer for having things
online in the next few days. Unfortunately the
Web developer is on holiday--you find a newly
hired guy to replace him. Would [E.sup.2]ML
support Web material development, and support
the new guy in understanding what you want to
do?
Would the [E.sup.2]ML documentation be useful    Checking implementation
for checking the implementation status of a      status
course?

scenario                         By evaluation

Team communication (1)           2.00
Compare design (2)               1.83
Consistency check (3)            1.75
Adaptation, new instructor (4)   1.75
Teaching novice designers (5)    1.75
Adaptation, new target (6)       1.67
Check implementation status (7)  1.58
Web material development (8)     1.42
Work overload (9)                1.27
Diagnostic for redesign (10)     1.17

Figure 6. Scenarios evaluation chart

Note: Table made from bar graph.

                                         evaluation

POSITIVE FEATURES
Can enhance the quality of instruction   1.00
Can support more challenging solutions   0.64
Can smoothen the design process          0.82

NEGATIVE FEATURES
Too large effort, no return              0.09
Has too many elements                    0.09
Is too complicated                       0.36

APPLICATION
Would be nice with an application        0.73

Figure 7. Statement results chart

Note: Table made from bar graph.