Online Academy: content validation through a juror process.

The Online Academy (HO29K73002) was funded by the Office of Special Education Programs (OSEP) to develop research-based online instructional modules in the content areas of reading, positive behavior support, and technology across the curriculum. Targeted to preservice teacher education programs in institutions of higher education (IHE), but also applicable to staff development, the modules were adopted for implementation by 170 institutions. This article describes the juror process employed by the Academy to identify the research base, frame the content maps, and validate the content. Sixteen national content experts served as jurors in this process. A total of 75 lessons in 22 content rich online modules were developed, each approximately equivalent to a two-semester credit course.

**********

Jl. of Technology and Teacher Education (2003) 11(3),443-454

BACKGROUND

The Office of Special Education Programs (OSEP) funded the Online Academy (onlineacademy.org) in 1997 to develop online modules for teacher education. The intent was to take advantage of the emerging Internet technology as a way to distribute instructional resources in teacher education. Access to resources trhough the Internet meant that the resources in the form of online modules would be made available anytime, anywhere. Online access also allowed for the modules to be developed for the students' use while at the same time providing resources for instructors. The modules developed by the Academy were designed for use in an interactive mode with instructors or as self-contained instructional units. They were structured to be used as independent instructional resources or used in part as supplemental resources in courses. All modules employed streaming media, and all of the elements within each module could also be printed as hard copies.

OSEP required the modules to be in the three content areas (i.e., reading, positive behavior supports (PBS), and technology in education) and to be research-based. This presented a serious challenge due to the variability of the research base across the content areas and the difficulty in validating content. In examining the content areas to be covered by the Academy, it was apparent that reading had a significant research base characterized by differing philosophical perspectives. PBS represented a much smaller, but significant research base cutting across several disciplines. Finally, technology in education is a relatively new area of research that can best be described as a literature base of best practices rather than a research base.

While literature review procedures were available for identifying the research applicable to practice, models were not available for validating the content once it was developed for online delivery. Developing and validating the content for each module was viewed as a major undertaking due to the breadth of the content. Thus, it was decided to combine the generation of content with the selection of research to be highlighted and the validation of content. A juror model involving content experts knowledgeable about research in the content areas was adopted. Specifically, the juror model, combined with staff writing teams, was employed to validate the content.

In searching for direction on how to employ a juror model in the content selection and validation process, the referee procedures applied by academic journals was first examined. The process of engaging experts in a field to review manuscripts based on criteria adopted by the publisher offered some guidance. However, because our goal was to engage jurors in a formative process to influence the development of content, render qualitative decisions on the effectiveness of research, and comment on the appropriateness of studies for elaboration in the instructional modules, the confidential nature of the referee process was considered too restrictive. However, it was believed that some version of the peer-review process popular in the scholarly press (Leslie, 1994) was essential to achieving our goals.

Having made a commitment to the use of content experts, we examined sources related to the application of this approach in the field of instructional design. Doerksen (2002) refers to three essential ways to develop content for educational media. These include: (a) teach multimedia and instructional designers to become subject matter experts; (b) teach subject matter experts to become instructional designers; and (c) team subject matter experts with instructional designers. The team approach, according to Doerksen, should lead to the most efficient content development process. Keppel (2001) published a study that explored the development, trial, and implementation of a process designed to assist instructional designers in eliciting and conceptualizing unfamiliar content from subject matter experts. Keppel also subscribed to the team approach because it is rare for one individual to have all the skills necessary to create and complete a multimedia project. The subject matter expert--or in this context, the juror--is an authority on a particular domain of the knowledge, who can provide a clear description and explanation of the content area, and also clarify and verify the content.

Lee and Owens (2000) suggested that validating content is best done in the design phase, using the expert review process. The main functions of the experts are to review the course materials and validate that there is congruence between the objectives, content, and test items. According to Lee and Owens, minimum validity is especially important when the course is used for certification of competence in a given subject.

Gruber (1994) used a juror model to identify and validate content and indicators of quality in international education programming. Vrasidas and Harris (1995) involved jurors in reviewing prototypes of hypermedia CD-ROM products. Their jurors included instructional designers, multimedia developers, graphic artists, and content experts. Hoover and Abhaya (1995), in applying the juror model in the development of computer-based instruction, found that it was important to clarify how much faith content experts have in instructional design. While the jurors with instructional designers, were not directly involved they were engaged in expanding the design for the online modules with particular attention being given to how the content would interface with the design. Maple (1994) examined the relationship of instructional designers and content experts and recommended simplifying language to eliminate jargon and focus on interpersonal skills. Since the emphasis was online instruction and at that time there was a lack of experience with regard to technology on the part of those who served as jurors, we clearly diminished the use of language relative to technology and focused on the content research and functionality of the technology.

METHODOLOGY

The methodology for moving research to practice required the development of processes for selecting research to be integrated into the online instructional modules and for validating content. This involved two primary strategies. The first strategy was to use content experts as jurors in the selection of research related to practice, validation of content for each content area, and development of content maps. The second strategy was to use staff writers to prepare the content according to specifications of the jurors and the production system that was used to transform content into operational modules for online delivery. These strategies were adopted to ensure that the most appropriate research was selected as the basis for the content and that the content written for inclusion in the modules was externally validated. It was also essential that the content be written in a form that met the format requirements of the production tool for online delivery. Each of the strategies will be described in more detail below.

Strategy I: The Juror Process

The juror process involved engaging nationally recognized experts in their respective content areas. Nominations were sought from colleagues nationally. Literature reviews were also conducted by Academy staff to identify individuals doing research in the content areas. In addition, consultation was sought from teacher education faculty nationally in the content areas and from other individuals considered to be knowledgeable in the field. An effort was made to achieve a representative group of jurors within each content area; however, priority was placed on expertise in the content and knowledge of related research. Personal involvement in research was also considered.

Membership on the juror teams included 18 individuals as illustrated in Table 1. The role classification of jurors pertained to their current positions. It should be noted however, that those identified as professors all had K-12 teaching experience and were engaged in research related to one of the three content areas. The research conducted by the researchers and professors involving K-12 students took place in controlled environments in educational settings.

The input of the jurors was central to the role of the writing teams. For most content areas, juror meetings were held twice a year during the term of the project. The initial meeting included an orientation to the Online Academy project, followed by an explanation of participants' role as jurors. The three juror teams (one for each content area) met as a group for the first session. In subsequent sessions an attempt was made for teams to meet individually, but at the same time. However, this did not always work as the priority was placed on full attendance within juror teams versus getting the teams to meet on the same date.

The typical meeting involved a brief update on the work of the Academy with most of the time committed to the work of the jurors in their respective content areas. Initially, the work focused on identifying research applicable to the respective content areas. The agenda then moved to providing input on content maps for module development and later to a review of the content for individual modules. As content was written and entered into the design tool, jurors had an opportunity to review it in the online module format. Finally, jurors were available on-call and for onsite consultations.

Juror responsibilities. The following is a listing of the major responsibilities assigned to the jurors to ensure selection of the most appropriate research and validation of the content.

1. Reviewing and advising on the research and content standards developed by the staff.

2. Setting criteria for selecting research and advising on the implementation of the criteria for selecting research studies or validated practices to be considered for inclusion in instructional modules.

3. Providing leadership in developing content maps for each module.

4. Reviewing module content during the development process.

5. Participating in the Alpha testing of the module design. (This was done mainly to orient the jurors to the design of the modules and demonstrate how students would use them.)

6. Problem solving with writers as needed.

7. Participating in the decision of when modules were to be considered in final form.

8. Providing input on beta testing procedures and sites.

Strategy II: The Team Writing Process

For each content area a writing team was appointed. Writing teams were comprised of at least one professional staff member and a varying number of graduate research assistants with experience in the specific content area. In addition, individuals nationally with content expertise were also contracted as writers when needed.

The writing team served as the primary point of contact for the jurors. Specifically, the writing team's responsibilities, in terms of the research and content validation, were to work closely with the juror team (in each respective content area) in the development of content. The staff set the agenda for each meeting and worked directly with the jurors as needed. In addition, the staff conducted traditional literature reviews to supplement the work of the jurors. Their primary responsibility was to translate the recommendations of the jurors on research and validated practices along with the findings from the literature reviews into first-draft content maps. They subsequently worked with the jurors to refine the content maps that served as the initial blue print for developing content for each module. Because the relationship between the writing teams and the jurors was an iterative one, the content maps were a work in progress during the early writing stages in each content area.

Following the development of content maps, the first step in writing was to be certain that all partners understood the instructional design and the specifications for writing content. This was quickly followed by the development of detailed outlines based on the content maps. When questions or problems arose relative to either the research or validated practice within a content area, the teams consulted the respective jurors. Although an effort was made to write the content for each module sequentially, in reality, two or three modules typically were being worked on at any given time during the writing stage.

RESULTS

The juror approach to content validation and development resulted in the development of 75 lessons and 22 modules, each including over 2000 text files and 500 graphic files. Each module involved approximately 12-14 clock hours of instruction. Because the juror model was so central to the Academy approach to module development, the process was integral to the outcomes of the Academy, and therefore the roles of most staff members.

The following discussion focuses on the juror model based on observations from staff and jurors relative to the process over two and a half years of content development. It concludes with a look at the writing team strategy.

Staff Observations

1. The strength of the juror strategy was the knowledge and insight the jurors brought to the process about the research base and its implications for the respective content areas.

2. Each group of jurors took its responsibilities seriously. Rarely was anyone absent from a meeting, as evidenced in a 90% percent rate of attendance at all meetings across the three teams of jurors. In addition, all jurors completed the three-year assignment with the project.

3. The two-day sessions held during the process of developing the content maps proved extremely important. The jurors took a while to develop a perspective of the scope of content to be covered by the modules before they could focus on the content for individual modules. In part, the extended sessions served to create a culture for the jurors and allowed them to learn each other's perspectives.

4. The content development process yielded content maps in a timely manner. However, it soon became apparent that it was necessary to view the development of content maps as works-in-progress, as the research review and content development process resulted in changes as content was written. Posting the content maps on a web site allowed jurors to provide feedback to writing teams on a continuous basis.

5. This juror model allowed for maximum input in the content development process. Consultation with individual jurors greatly enhanced decisions on content changes and shifts in the number of lessons within a module and the number and titles of modules.

6. Some jurors elected to personally write modules, lessons or parts of lessons. This was not required or anticipated. However, it reflected the level of interest in the content and the desire to be part of the writing process. When a juror assumed a writing responsibility, a member of the respective writing team took on the role of a content manager, working closely with the juror (writer) to ensure that the content specifications of the design tool were met.

7. Jurors varied in their knowledge of and interest in technology (except for those in the technology in education content area). However, their interest in the content and in moving research to practice was uniform. When modules were made available for review on the web site, several jurors preferred the online format, whereas others preferred to review the content in hard copy.

8. Individual consultations occurring outside of juror meetings were more frequent with jurors who elected to participate in the writing process. Consultations resulted from the initiative of jurors as well as in response to requests from writing team members. The value of the individual consultations was on par with the juror sessions. The difference was that the consultation input tended to relate to very specific topics, module features, selected resources, or the presentation of content, whereas the group sessions were oriented more toward determining content and integrating research.

9. Jurors were very open and candid in their interactions and consultation. At times they disagreed among themselves, but were always able to reach consensus to fulfill their roles.

Juror Observations

At the end of the three-year project, jurors' observations were solicited on the effectiveness of the juror model in moving research to practice and validating content for online modules. An evaluation instrument was placed online. Each juror had an opportunity to respond to specific queries and to offer comments. Responses were anonymous and aggregated across content areas; as a result, they reflect on the process, not the work of specific juror teams.

The following is the list of items in rank order, with the highest ranked first. (There was no difference in how the first four items were ranked.)

1. The Academy staff was responsive to my inquiries and/or suggestions.

2. The juror process is effective as employed by the Academy.

3. I had a positive influence on the content area.

4. The meetings were well planned and productive.

5. My input was respected.

6. There was sufficient opportunity for me to provide input to the writing teams.

7. My responsibility as a juror was sufficiently clear.

8. There was a good balance of expertise among the jurors on my team.

9. If asked to serve in a similar role in the future, I would be willing to do so if my schedule permitted.

10. The juror method is an effective way to select research for transfer to practice.

11. The juror board of which I was a member worked well together.

Jurors were also allowed to submit anonymous comments in response to the survey.

The following comments were representative of those submitted by jurors.

1. This was a monumental project that broke new ground.

2. This is an effective way to engage experts in a process that results in product development.

3. Need to actively solicit involvement of culturally and linguistically diverse participants.

4. More communication outside of meetings, including a listserv and more meetings, would have been helpful.

5. The strength was in the wide variety of expertise and perspectives brought to bear on the issues.

6. More clarity on roles and relationships with writing teams would be beneficial.

7. More input from stakeholders to supplement the input from jurors would help.

8. Once writing began there could have been more emphasis on the nuts and bolts of developing the modules.

Writing Team Strategy

The strength of the writing team strategy in the research/validation process was the consistency of staffing across the three years of the project. That is, while there were changes among staff, the lead staff person on each writing team remained consistent during both the writing and production phases of the Academy.

The structure of the teams included a coordinator, a lead writer, and staff writers. Many of the writing staff were graduate research assistants experienced in the content area. Most members of the writing staff at some time were involved in a review of research on a module topic. Staff members interacted with jurors during onsite meetings. However, the lead writer was the key person who routinely communicated with jurors.

While the research-to-practice content review by the staff was an iterative process with the jurors, the teams also pursued independent literature search initiatives. This was essential to maximize efficient use of time. The sequencing and time required for module development allowed for effective communication between the content-area staff and the jurors during the research review stages. Due to the number of modules involved for each content area, the research review process was a constant activity throughout the three years of the project.

SUMMARY OF LESSONS LEARNED

The core staff, including coordinators and lead writers, met weekly to coordinate the activities of the Academy. This provided an effective mechanism for monitoring the jury process. Particular attention was given to sharing anecdotal experiences and observations to improve the process as the Academy anticipated doing further module development in the future.

Lessons learned that should be considered in modifying the process for future applications of the juror model include:

1. Centralize coordination of juror teams in one person to capitalize on the experiences of each team and to ensure that processes and communications are similar across teams.

2. Increase opportunities for communications among jurors (e.g., more frequent meetings, use of a listserv, teleconferences and more precise guidelines for how they can communicate with team leaders and staff).

3. Engage jurors from all teams in an orientation/training session at the beginning to share the Academy process and the lessons learned.

4. Be more aggressive in seeking diversity among jurors.

5. Create links with input from stakeholders, where appropriate, and ensure access to the perspectives of the stakeholders and jurors.

6. Provide jurors early access to the internal web site where modules are posted during the formative process.

7. Be more assertive in exploring jurors' interest in assuming responsibilities for writing modules or parts of modules.

8. Encourage jurors to view content in the online version.

One of the lessons learned that was clearly evident across the three groups of jurors was that the collegial process works. There were philosophical differences among jurors and each had his/her preference for research studies, yet the number of studies considered and the amount of content to be included had to be limited. Each team of jurors, after full discourse, was able to agree on what was to be included. It is difficult to attribute this to any particular event. It may have been due to the national nomination process, the universal commitment to the goals of the project, the fact that the jurors knew each other and/or their work, or just good luck.

The use of jurors was considered critical to meeting the commitment of developing research-based modules with validated content. The richness of the content can also be attributed to the input of the jurors and their collaborative work with the writing teams. It was believed that the adoption of the modules by over 170 universities can be attributed in part to the effectiveness of the juror process. The juror process was viewed as one of the central elements of the Academy along with the tool design, content generation, production, Beta testing, and implementation processes. The intent was to continue refining the juror process in subsequent projects where the output is research-based and content-rich online instruction.

Table 1

Composition of Juror Teams

Professors                   13

Researchers (non faculty)     2

Teachers                      2

Other                         1

Female                        7

Male                         11

Acknowledgments

Preparation of this article was supported in part by the Office of Special Education Programs and Grant #HO29K73002.

We acknowledge the contributions of the Online Academy staff, the individuals who served on the various advisory boards, jurors and the many institutions, and individuals who contributed to the evaluation of the online modules and the production tool. Appreciation is expressed to the Center for Research on Learning and the Information & Telecommunications Technology Center whose collaboration created the e-Learning Design Lab in which this work is being conducted.

References

Doerksen, T. (2002). Development approaches. In H.H. Adelsberger, B. Collis, & J.M. Pawlowski, (Eds.), Handbook on information technologies for education and training (pp. 181-198). Berlin: Springer-Verlag.

Gruber, L.J. (1994). Beyond borders: A validation of the areas and indicators of quality of international education programming in U.S. twoyear colleges. Unpublished doctoral dissertation, Northern Illinois University, Illinois.

Hoover, S.J., & Abhaya, P.S. (1995, month). Instructional design theory and scientific content for higher education. Proceedings of the 1995 Annual National Convention of the Association for Educational Communications and Technology (AECT), Anaheim, CA.

Keppell, M. (2001). Optimizing instructional designer-subject matter expert communication in the design and development of multimedia projects. Journal of Interactive Learning Research, 12 (2/3), 205-223.

Lee, W.W. & Owens, D.L. (2000). Multimedia-based instructional design: Computer-based training, web-based training, distance broadcast training. San Francisco: Jossey-Bass.

Leslie, J. (1994). Goodbye, Gutenberg: Pixelating peer review is revolutionizing scholarly journals. WIRED, 2, 10.

Maple, R. J. (1994). "Well, you're the CE ... I'm the ID ..." describing your role--and selling your worth--to content experts. Performance and Instruction, 33(8), 36-40.

Vrasidas, C., & Harris, B.R. (1995, October). The development of an interactive multimedia CD-ROM: Lessons learned. Selected Readings from the Annual Conference of the International Visual Literacy Association, Eyes on the Future: Converging Images, Ideas, and Instruction, Chicago, IL.

EDWARD L. MEYEN AND YVONNE N. BUI 

The University of Kansas 

USA

[e-mail]