Virtual design based research.

Abstract

Instructional materials for the online course may be designed with rich multimedia content and innovative approaches. Student engagement with these materials is unseen and left to be inferred. Design based research is an approach used to study intervention in the context of an authentic classroom. It can be applied to the online course to study the impact of instructional design decisions and the effectiveness of multimedia integration. The technology used to support the online course can be used to collect usage data and participant evaluations.

Introduction

Online instruction can seem like an experience in teaching to the great void. Course content and assignments are designed, developed, and posted online for unseen students who are out there somewhere in cyberspace. An instructor may devote long hours crafting multimedia rich course content only to wonder if students are using and benefiting from those materials. Student engagement with course content can be a mystery even in traditional face to face classrooms. One of the benefits of online instruction is that the course materials are posted on Web servers or course management systems and usage tracking is possible. In a traditional course it is difficult to determine if a student picks up a textbook to read it, but in the online course the mouse click to open the Web page containing an online reading assignment or multimedia presentation can be recorded. Additional data can be collected in the online course through anonymous online evaluation surveys. Data can be imported directly into spreadsheets or databases thus eliminating the need to transfer information from paper survey instruments. The usage and evaluation data can be analyzed by the online instructor to improve course design in a manner consistent with existing instructional design models (Dick, Carey, & Carey, 2005; Gustafson & Branch, 2007). The formative and summative evaluations used as part of the instructional design process are valuable for revision of specific designs created for defined groups of learners. A shift toward a research oriented focus for instructional design is necessary in order to move into the broader scope of learning theory.

Instructional Engineering for Online Learning

Instructional design is oriented toward the practical goal of developing effective teaching and training solutions. A transition can readily be made from instructional design to an instructional engineering approach by changing the emphasis to one of design centered research. The term instructional engineering is one possible name for the partnership between instructional design and research. This idea was suggested by Brown (1992) when describing how she engineered innovative educational environments while simultaneously conducting experimental studies to learn how they functioned in an authentic classroom. The term design experiment has been used by Brown and others when discussing the strategy of designing an instructional intervention to be tested through research in an authentic classroom context (Brown, 1992; Collins, 1999). Alternative names for the methodology include design research (Collins, Joseph, & Bielaczyc, 2004; Edelson, 2002), and design based research (Barab & Squire, 2004; The Design-Based Research Collective [TDBRC], 2003). These terms are often used interchangeably.

An instructional engineering approach can be applied to the design and study of virtual learning environments. From this perspective the online course is treated like a product to be designed and tested through a design based research methodology. This process has been described as being similar to product design engineering (Zaritsky, Kelly, Flowers, Rogers, & O'Neill, 2003). The instructional engineering process might begin with conceptualization of the product (i.e. online course) based on relevant overarching theory. This would be followed by design and development of a prototype. Next, the prototype is tested and data is collected to inform revision of the design. This creates an iterative process of conceptualization, design, test, and redesign in order to improve the product, which in this case would be the online course. This process is essentially what design based research is all about. An innovation is designed and studied within the context of the learning environment it was intended for. One benefit is that all systemic variables that might affect success of the design are present. A design that works well within the controlled and protected experimental laboratory may fail after encountering these extraneous variables. Because of this, design based research can be used to complement experimental research. Experimental research is used to investigate isolated variables and design based research is used to investigate a system of variables.

An example of the interplay between experimental and design based research is found in the area of multimedia learning. Many experimental studies have been conducted to examine how learning is affected by the use of multimedia for instruction. For example, Mayer (2001) conducted a series of experimental studies over more than a decade to develop a set of principles for multimedia learning. These studies were conducted in a face to face controlled laboratory setting. There is considerable evidence supporting the effectiveness of multimedia as an instructional tool when its design is based on the principles described by Mayer. The next question to consider is whether or not the multimedia learning principles remain effective when applied to a new context such as an uncontrolled and messy classroom situation. The online learning environment adds additional complexities due to the reliance on Web technologies and the unseen use of multimedia by students. There is no chance of the principles being effective if the technology fails to work. Furthermore, if students do not use the multimedia then the benefits are forfeited and instructor time was invested for nothing when creating it. If the technology works and students use the multimedia then there still may be design issues that minimize its effectiveness. Problems such as these tend to surface when integrating multimedia in the functioning online course.

The Virtual Research Laboratory

The online course can be structured as a type of virtual research laboratory where new designs for instruction may be simultaneously implemented and tested. Design based research promotes the combination of design, research, and practice. However, the guidelines for conducting this type of study are vague and still open to interpretation. Design based research is still emerging and borrows from other methodologies such as action research and formative evaluation (Wang & Hannafin, 2005). What differentiates it most from other forms of research is the emphasis on theory development derived from studying learning designs as they unfold under the influence of contextual variables (Barab & Squire, 2004; TDBRC, 2003). In an online course multimedia learning theories could guide the design of instructional content which is then tested in the context of the virtual classroom. When the results of similar studies set in different contexts are combined a context domain or profile of learning environments where a design has been tested successfully is established. For multimedia learning this may included a range of contexts such as the experimental lab, face to face classroom, and online classroom. In this way theory is shown to either work well or fail in an array of learning environments.

Since design based research is emerging there is some question about what it really means to do this type of study. A number of articles and book chapters on the topic provide a sense of the general features of the methodology. Many of these have been published in recent years indicating an increased level of interest. At least three special issues of peer reviewed journals have featured the topic of design based research or design experiments. These include an issue of Educational Researcher in 2003, and two special issues in 2004 one of which appeared in The Journal of the Learning Sciences and the other in Educational Psychologist. The methodology has found its way into grant funded research projects as well. Approximately 20 percent of grants awarded between 1996 and 1998 by the Division of Research, Evaluation, and Communication at the National Science Foundation may be categorized as design experiments (Suter & Fretchtling, 2000).

Much of the literature about design based research is focused on what it is or how it might be characterized. A few case studies have been written to explain how it has been applied in actual research (diSessa & Cobb, 2004; Joseph, 2004). These studies are set in the context of face to face classrooms. A gap in the literature exists for design based research studies that are set within the context of a virtual classroom. It has been argued that design based research has potential for the study of technology-enhanced learning environments (Wang & Hannafin, 2005). Online instruction is still evolving and much needs to be learned about effective practice. Design based research is appropriate for studying online learning environments and the impact of instruction designed for them.

A starting point in the process of developing a virtual design based research study is to define a workable process. In general, design based research involves iterative cycles of design and implementation (Edelson, 2002). Data collected during each cycle is used to make revisions for the next cycle and develop, refine, or extend theory. Cycles are classified as macrocycles and microcycles (Jonassen, Cernusca, & Ionas, 2007). A macrocycle is a long phase such as a semester and the microcycle is a shorter phase such as a day or week of class. During microcycles adjustments can be made to the design or strategies used for collecting and evaluating data. In a study of multimedia learning within an online course the microcycle would be the individual assignment phase that occurs daily or weekly. Problems with the multimedia design can be corrected during each microcycle as the course progresses through the semester long macrocycle. After the course has been completed additional modifications can be made before repeating the use of the multimedia in a new session of the course. These modifications are made after analyzing the data and reconsidering the theories driving the design of the multimedia.

The virtual design based research study may be structured in a set of systematic steps that can be followed during each microcycle as well as the macrocycle. Continuing with the example of the multimedia enhanced online course these steps are conceptualized as follows.

* Craft the Design: Develop the overall framework for the entire course including the syllabus, course schedule, and learning goals. For each assignment develop multimedia presentations and simulations that are designed to meet the course goals. During the design process use a combination of instructional message design and multimedia learning principles to guide development of the design (Clark, 2005; Fleming & Levie, 1993; Mayer, 2001).

* Test the Design: Implement the multimedia while the course is in progress. Test the multimedia using multiple Web browsers and computers to assure that the technology is functioning properly. After the multimedia is made available to students collect usage data through course management statistics and server side scripts. Use online forms to collect anonymous evaluation data from the course participants. The anonymity will protect students from coercion due to grade penalties and produce honest feedback.

* Analyze the Data: Feedback from the evaluation surveys will provide information about the perceived value of the multimedia for learning. Comments submitted by students who complete the evaluations provide insight into the nature of technical problems or misunderstanding about the multimedia content. Usage data will reveal patterns of access such as frequency of use, peak usage days, or types of Web browsers being used to access the multimedia.

* Build Theory: After analyzing the data determine if there is evidence to support revision or extension of the theories applied during design of the multimedia. For example, if the multimedia is rated poorly by students is it because of a flaw in the design or the theories guiding the design? If students are not using the multimedia is it because of technical problems or is it a matter of motivation? These questions can be studied in the next cycle of multimedia integration.

* Revise and Retest: Revise the design of the multimedia to improve it and also to continue testing theory. For example, the data may indicate that motivation issues are affecting frequency of use. In the next cycle of implementation adjustments can be made to the design to see if the multimedia is used more often.

The previous sequence of steps demonstrate a practical example of one way that a virtual design based research study might be structured. Although multimedia learning was discussed in these steps the process is not limited to that particular focus. Any number of designs for learning could be studied through design based research in the virtual laboratory of the online course.

Conclusions

The number of programs offering virtual instruction has grown to the point where online education is now being discussed as having entered the mainstream (Allen & Seaman, 2005). There is high demand for online courses and educational organizations are actively developing or expanding options. The challenge ahead is not only meeting the demand, but also developing a deeper understanding of online teaching and learning. There is still much to learn about best practice across the landscape of possible online course offerings. Through design based research innovations in online instruction may be designed and simultaneously tested in the real-world online classroom. The technology supporting the online course provides a mechanism for obtaining data to inform designers, researchers, and practitioners about the unseen world of student learning online. Next steps for a research agenda in online education include further development of vocabulary and methodological processes that will better support meaningful dialogue about virtual design based research. Doing this will lead to the evolution of methodologies appropriate for integration of design and research in the online course.

References

Allen, I. E., & Seaman, J. (2005). Growing by degrees: Online education in the United States, 2005. Neeham and Wellesley, MA: The Sloan Consortium.

Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. Journal of the Learning Sciences, 13(1), 1-14.

Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141-178.

Clark, R. C. (2005). Multimedia learning in e-courses. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 589-616). New York: Cambridge University Press.

Collins, A. (1999). The changing infrastructure of educational research. In E. C. Lagemann & L. S. Shulman (Eds.), Issues in Education Research: Problems and Possibilities (pp. 289-298). San Francisco: Jossey-Bass.

Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15-42.

Dick, W., Carey, L., & Carey, J. (2005). The systematic design of instruction (6th ed.). New York: Pearson.

diSessa, A. A., & Cobb, P. (2004). Ontological innovation and the role of theory in design experiments. Journal of the Learning Sciences, 13(1), 77-103.

Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105-121.

Fleming, M., & Levie, W. H. (Eds.). (1993). Instructional message design (2nd ed.). Englewood Cliffs, NJ: Educational Technology Publications.

Gustafson, K. L., & Branch, R. M. (2007). What is instructional design? In R. A. Reiser & J. V. Dempsey (Eds.), Trends and issues in instructional design and technology (2nd ed., pp. 10-16). Upper Saddle River, NJ: Pearson.

Jonassen, D., Cemusca, D., & Ionas, G. (2007). Constructivism and instructional design: The emergence of the learning sciences and design research. In R. A. Reiser & J. V. Dempsey (Eds.), Trends and issues in instructional design and technology (2nd ed., pp. 45-52). Upper Saddle River, NJ: Pearson.

Joseph, D. (2004). The practice of design-based research: Uncovering the interplay between design, research, and the real-world context. Educational Psychologist, 39(4), 235-242.

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.

Suter, L. E., & Fretchtling, J. (2000). Guiding principles for mathematics and science education research methods: Report of a workshop (No. NSF 00-113): National Science Foundation.

The Design Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8.

Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research & Development, 53(4), 5-23.

Zaritsky, R., Kelly, A. E., Flowers, W., Rogers, E., & O'Neill, P. (2003). Clinical design sciences: A view from sister design efforts. Educational Researcher, 32(1), 32-34.

Chareen Snelson, Boise State University, Boise, Idaho 

Chareen Snelson, Ed.D., is an Assistant Professor of Educational Technology in the College of Education. She designs and teaches multimedia enhanced online courses.