Project MERLOT: bringing peer review to web-based educational resources.

The unprecedented growth of the World Wide Web has resulted in a profusion of educational resources. The challenge for faculty is finding these resources and integrating them into their instruction. Even after the resource is found, the instructor must assess the effectiveness of the resource. As the number of educational web sites mount into the millions, this task is becoming increasingly difficult.

To address the problems of finding instructional sites, evaluating their educational value, determining their accuracy and assisting professors to integrate the resource in a learning environment, the Multimedia Educational Repository for Learning and On-line Teaching (MERLOT) was established. The MERLOT virtual repository (www.merlot.org) of learning objects is hosted by the California State University Center for Distributed Learning. The project also provides a peer review process to ensure the quality of the educational resource.

**********

The exponential growth of the World Wide Web has resulted in a profusion of educational resources that are potentially useful in higher education. The challenge for faculty interested in using web-based resources as an important aspect of their instruction is finding these resources and integrating them into their instruction. Even after the resource is found, the instructor must assess the effectiveness of the resource. As the number of educational web sites mount into the millions, this task is becoming increasingly difficult.

The unparalleled growth of the Web is partly due to the fact is that it is so easy and inexpensive to develop a web site. With the aid of inexpensive or free web editors, creating a web site is almost as easy as using a word processor. There are also a myriad of companies that will host a web site for very little or no money. Since it is so cheap and easy to establish a web presence, the number of web sites has proliferated. While using the traditional search engines can be useful in finding educational sites related to a specific topic, experienced web users know how difficult it can be to find relevant information. Search engine may find a few million sites it deems relevant, but it does not ensure the quality of the information on these web sites. Educators have a particular responsibility to ensure that web resources that they use in their courses are authoritative.

The concept of authoritative asks the question, "Who says this material is accurate?" By knowing the background of the authoritative source of a work, one can judge how much one trusts it. For example, if one were to access the Microsoft[R] knowledge base (Technet Online, 2001) to look up information on installing a new Windows XP driver, one would tend to trust the information. When students access a professor's course web site, they would know that the professor is the authority behind the information on the site. The professor's students would know that they need to believe what they read, at least for the rest of the semester. Since these pages are probably posted on a university's server, the reputation of the institution also gives it additional authority. However, until recently, there has not been any objective review of higher education web resources available.

Even after finding an educational resource and determining its accuracy, the potential user must determine if the resource is effective in teaching or demonstrating the concept. A web page could be both relevant to your topic and accurate, but may not be useful by the target audience you want to use it with. If it is pure text with no interactivity, has inappropriate vocabulary, or is simply ineffective instruction, it may not be useful. To address the problems of finding instructional sites, determining their accuracy, and determining how to integrate the resource in a learning environment, the Multimedia Educational Repository for Learning and On-line Teaching (MERLOT) was established. The MERLOT virtual repository (www.merlot.org) of instructional materials is hosted by the California State University Center for Distributed Learning. As the project evolved, a peer review process was initiated to "improve the effectiveness of teaching and learning by expanding the quantity and quality of peer-reviewed online learning materials" (University of Michigan, 2000). The balance of this article discusses both MERLOT as a virtual repository of web-based educational resources and as a source of peer reviewed resources.

MERLOT AS A VIRTUAL REPOSITORY OF EDUCATION RESOURCES

The MERLOT community has established and maintained a virtual collection of educational resources at its web site. Membership in the MERLOT community is open to any interested educator and there is no cost for membership. Once you have registered as a member on the MERLOT web site, you can contribute to the MERLOT community by posting links to web-based materials or by providing user comments to sites already posted. MERLOT members also receive periodic updates about MERLOT activities and projects.

Experienced web users know that there are several ways to find information on a web site. When one was interested in searching for broad topics, a web index is likely to be useful. These indexes are useful for browsing, using hyperlinks to make it easy to navigate through a web site. However, when you are looking for something more specific, a powerful search engine may be more useful. These search engines allow for complex searches using Boolean logic and sometimes even use artificial intelligence to find information. To accommodate both of these approaches, the educational resources, called learning objects can be accessed by either browsing through the materials by broad subject areas or using a powerful search engine. After a brief overview of learning objects, a further discussion of navigating the MERLOT site will be presented.

LEARNING OBJECTS

Learning objects are modular digital resources that are used to support instruction. By modular, we mean that the educational content is broken down into a sufficiently small chunk so it can be used in a variety of educational settings. Conceptually, learning objects are closely tied with the philosophical approach used in object oriented computer programming (Wiley, 2000).

Just like the object oriented approach revolutionized computer programming, learning objects provide a new way of conceptualizing learning content. Learning objects are small, self-contained, and can be reused with little or no modification. Most learning objects are designed to take between 12-15 minutes to complete and must contain at least an objective, an activity, and an assessment. The objective is the instructional goal of the learning object. The learning activity is actual interaction that the learner engages in and the assessment insures subject matter mastery.

One important aspect of learning objects is that they must be metatagged so that they can be stored in and retrieved from a database. The importance of having the learning objects searchable cannot be overstated. The MERLOT site alone contains over 10,000 learning objects in 14 disciplines. To summarize, the Wisconsin Online Resource Center noted that learning objects must be "Just enough, just in time, [and] just for you." This means that learning objects are modular (just enough), searchable (just in time) and customizable (just for you) (Wisconsin Online Resource Center, 2005).

BROWSING THE MERLOT SITE

As shown on the MERLOT home page (Figure 1), the learning objects are categorized by seven subject areas (Arts, Business, Education, Humanities, Mathematics, Science and Technology, Social Science).

[FIGURE 1 OMITTED]

These broad subject areas enable the user to access a web index for the selected topic. For example, selecting Science and Technology produces the following index (Figure 2).

[FIGURE 2 OMITTED]

The display now shows all of the subcategories of the Science and Technology Index. You can continue drilling down into these subject areas to browse the appropriate resources. As noted above, MERLOT provides two methods of browsing. The second method is organized by specific academic disciplines rather than subject matter. The Discipline Community Index is accessed from the main MERLOT menu (see Figure 1) by clicking on Communities. This displays the MERLOT Community web sites (Figure 3).

[FIGURE 3 OMITTED]

The screen now displays the various MERLOT Discipline Communities. By browsing further, you can find learning object indexed by these disciplines. For example, selecting the Information Technology discipline displays the subcategories of the discipline (Figure 4).

[FIGURE 4 OMITTED]

While having two indexes to the site may seem redundant, they provide two easy approaches to browsing the MERLOT site. They are both excellent ways to become familiar with the variety of learning resources available. However, as noted above, sometimes a powerful search engine is better suited for the task.

SEARCHING THE MERLOT SITE

The MERLOT search engine provides for both simple string searches and advanced searches that allow you to specify the fields you want searched. The advanced search facility also allows the use of Boolean (AND/OR) operators to refine the search. As web users know, sometimes an initial search is too broad and returns too many hits. To address this problem, the site also has an advanced subsearch capability that allows you to narrow the results of a search by adding additional conditions.

These searching and browsing capacities alone make MERLOT a valuable source for finding educational learning objects. However, as the project developed, it became obvious that MERLOT could perform an additional valuable service by providing some means of indicating the educational quality of the web sites. In 1999, a rigorous peer review process was initiated (MERLOT Room History Page, 2005. While only a small percent of the web sites on MERLOT have currently been peer reviewed, the number is constantly being increased. This will make MERLOT an even more valuable resource in locating and using quality, authoritative learning objects.

PEER REVIEW

Traditionally the quality of published educational materials has been ensured by the publisher of the material. Publishers use processes including peer review, editorial assistance, and other procedures to ensure that published materials are authoritative and up to scholarly editorial standards. Because publishing on the web is inexpensive, all of the expertise involved in the publishing process above the author is often eliminated. This means that the author, who sometimes isn't even identified, becomes the sole authority for the information on the site. The MERLOT community decided to implement a peer review process for learning objects to ensure that they are well designed and contain accurate information.

To develop a scholarly method for peer review, three models of academic review were considered: the peer review process used by journals, the process used in reviewing published materials, and the methods used for prepublication reviews. While all of these methods are scholarly, they serve different purposes. As a result, the review methods are different.

Scholarly journals require the author to submit previously unpublished materials for review. In general, authors are expected to have knowledge of journal's standards and must submit the material in a specific format. It is incumbent upon the author to follow the journal's style and subject area. The article is generally sent out to several scholars in the field for review. The more scholarly journals perform a blind review to ensure fairness in their acceptance policy. The journal editor will then read the reviews and make a decision on the fate of the work. The editor may accept, reject, accept with revisions, or reject with recommendations.

Book companies use a prepublication review to both determine if a manuscript is worthy of being published and how it can be improved. As such, it is more formative in nature. The reviewers are generally professionals in a field who are given a small honorarium for the review. The reviews are usually given to the author along with "suggestions" from the publisher.

After a work has been published, the editor of a professional journal may wish to have it reviewed. The editor selects both the reviewers and the book to be reviewed. Obviously, this type of review is for a different purpose. It is to evaluate the quality of an already published work rather than determining if something should be published.

Despite these differences, all of these models of scholarly peer review processes follow similar general processes to assure quality. For example, all of these publications establish evaluation standards and communicate these standards to the academic community. Reviewers then apply the evaluation standards to determine the worth of the material and write a recommendation regarding the material. The model developed by MERLOT is a synthesis of these models.

THE MERLOT PEER REVIEW PROCESS

To help higher education faculty to decide if a learning object is appropriate for their courses, MERLOT provides a structured peer review process. To ensure the validity of the reviews, they are all performed by experts in the academic field of the topic being reviewed, not by instructional technology experts. (Unless, of course, the academic topic being reviewed is instructional technology.) A learning object by be submitted to the MERLOT website by any member of the community. In some cases, materials are submitted by the author or creator of the resource but it may be submitted by any member who finds a useful resource they wish to share with others. Material submitted to MERLOT is first examined to determine its priority for being reviewed. In the MERLOT vernacular, this process is called "triage" and is designed to ensure that the highest quality materials are reviewed first. Materials that are accepted for review tag are those that received high marks in the triage process (Merlot Peer Review) and are assigned to the appropriate Editorial Board.

All MERLOT Editorial Board members are faculty with a record of expertise in the scholarship of their field, excellence in teaching and experience in using technology in teaching and learning In addition, Editorial Board members have connections to professional organizations in their discipline and experience in conducting peer reviews of online learning materials. Currently there are 14 discipline-based editorial boards. Each Editorial Board consists of and editor along with professional and volunteer Assistant Editors. Each Board is responsible for expanding and managing the collection of online learning materials in its discipline as well as educating and reaching out to the community of educators in its discipline. They are also responsible for implementing and managing the peer review process and for recruiting and training peer reviewers. The Editorial Board then assigns the resource for peer review. All reviewers are conducted by at least two professors that use web-based resources in teaching their academic discipline. The selected reviewers compose a composite review that is posted to the MERLOT website. If the reviewers do not agree on the value of the resource, the Editorial Board may send it to a third reviewer.

It should be noted that MERLOT is not the only guide to academic web sites containing learning objects. A fairly comprehensive directory of learning resources is maintained at the University of Texas, San Antonio (UTSA Learning Object Repositories, 2005). One of the most well know of these repositories is the Apple Learning Interchange (ALI) web site which contains a large number of learning objects along with many other educational resources (Apple Learning Interchange, 2005). While it is true that many scholarly associations and other groups have developed guides to course materials for their specific subjects, these sites generally do not assign ratings to resources (Young, 2000). This distinction makes MERLOT unique.

CONCLUSION

Historically, college professors are rewarded for their scholarly activities. A professor that publishes a book with a respected publisher is able to use that scholarly accomplishment for promotion and tenure. Unfortunately, if one chooses to publish electronically, it may be difficult to get the proper credit. The reason for this is partly due to the fact that the traditional scholarly processes used to validate a work have not been in place for the electronic media. MERLOT offers a solution to this problem. "If you spend a lot of time working on a module, then you should get some credit for it," says Cathy Owens Swift, a professor of marketing at Georgia Southern University who is one of MERLOT's reviewers. "People spend a lot of time developing modules, but nobody else ever sees them except their students." The MERLOT community is exploring other methods of recognizing professors who have developed excellent sites, including formal notification to their department heads (Young, 2000).

References

Apple Learning Interchange. Retrieved September 16, 2005, from http://ali.apple.com/

MERLOT Home Page. Retrieved September 16, 2005, from www.merlot.org

MERLOT Room History Page. A brief and simplified history of MERLOT. Retrieved September 16, 2005, from http://taste.merlot.org/history/history.html

MERLOT Peer Review Home Page. Retrieved September 16, 2005, from http://www.merlot.org/home/PeerReview.po

Technet Online (2001). Microsoft knowledgebase. Retrieved September 16, 2005, from http://microsoft.com/technet/default.asp

UTSA Learning Object Repositories. Retrieved September 16, 2005, from http://elearning.utsa.edu/guides/LO-repositories.htm

University of Michigan News and Information Services. (2000, August 23). Online instructional materials offered through new partnership. Retrieved September 16, 2005, from http://www.umich.edu/~newsinfo/Releases/2000/Aug00/r082300a.html

Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy, Retrieved September 16, 2005, from http://reusability.org/read/chapters/wiley.doc

Wisconsin Online Resource Center (2005). On-line learning object repository. Retrieved September 16, 2005, from http://www.wisconline.com/

Young, J. R. (2000, June 1). MERLOT project brings peer review to web materials for teaching. Retrieved September 16, 2005, from http: chronicle.com/free/2000/06/2000060101u.htm

RALPH CAFOLLA

Florida Atlantic University 

Hollywood, FL USA

[e-mail]