WebCT surveys: opportunities and challenges.

Abstract

Course management systems (CMSs) provide opportunities for instructors to create and implement web-based surveys, but some obstacles arise related to software compatibility issues and implementation challenges. Many of these problems can be avoided and others overcome through creative solutions.

Introduction

Since the dawn of the World Wide Web a decade ago, interest in web-based survey use has grown steadily. Research is still in the early stages of tapping the potential and discovering the limitations of using this medium for survey research. Nevertheless, the literature is rife with articles extolling the virtues of web-based surveys. Other researchers take a more cautious approach identifying problems with using this medium for survey research. One area of limited research that warrants further attention is the implementation of surveys in web-based course management systems (CMSs). This paper describes some of the implications manifested with the use of a WebCT survey during a study of web-based instruction and describes the solutions to these issues within one web-based CMS (WebCT).

CMSs are web-based software programs that provide an authoring interface and tools that allow instructors to create, facilitate and manage a course online without requiring knowledge of programming or web-development expertise. The popularity and proliferation of web-based CMSs have increased the use of surveys integrated within them. Regrettably, a thorough understanding of the implications and problems of surveys within this context has not accompanied this trend. One popular CMS used by many institutions in the U.S. is WebCT (www.webct.com). WebCT provides instructors more than 30 tools (including a survey tool) to help them create a web-based password-protected environment for their students. Because of the nature of this web-based context, some issues associated with web-based surveys identified in the literature are non-existent or minimized with the use of WebCT. The survey tool in WebCT assists an instructor in creating and implementing a web-based survey. This tool permits a number of options and capabilities such as providing anonymity for students participating in the survey, importing questions, permitting six types of question formats, integrating video and audio, tracking of non-participants, tracking of time spent on each question and on each survey, limiting participants to one survey attempt, automatically setting up a release date and end date of survey, and exporting of data to statistical packages. In spite of these benefits, the implementation of the survey tool in WebCT is not without challenges and implications that have not been widely documented in the literature.

Literature Review

Advantages of using web-based surveys include the ability to process results without separate data entry, cost savings, speed of processing and higher response rates if they are combined with other survey modes (Fricker & Schonlau, 2002) and the ability to observe participants' patterns of responding (Nichols & Sedivi, 1998). More technical involvement is needed in implementing and administrating a web-based survey compared to either a mail or telephone survey, but the electronic automation of a survey can save time during various stages of the survey process. Web-based surveys are automated through four processes consisting of 1) construction and format of the survey in digital format, 2) management of the survey, 3) download of the resulting survey data, and 4) the processing and analysis of the data. Technical issues often accompany electronic automation, and web-based surveys are no exception. Technical issues related to web-based surveys include the setup of the survey, navigation from question to question, and the compatibility of the survey with various browsers and hardware and software configurations (Schonlau et al., 2002).

The use of technology can greatly enhance the format, experience and complexity of web-based surveys by providing capabilities that do not exist in other survey formats. Web-based surveys can make use of web-coding (HTML, XML, DHTML, etc.), tables, a variety of colors and dynamic, multimedia, and interactive features (such as graphics, animation, quick-time movies, java-scripts, java-applets, audio) to enhance the survey experience in the hopes of getting more participants to respond to survey questions (Dillman et al., 1998a). But in order to access and complete these web-based surveys, participants must have Internet access and a minimum speed connection, a basic level of computer literacy, as well as minimum standards of computer hardware/software configuration and compatibility (Dillman et al., 1998b). Even with Internet access, many users are unaware of their computer configurations and do not have the basic computer literacy to complete the survey (Nichols and Sedivi, 1998). Web-based surveys that incorporate advanced design and programming features generally involve more time to display content and process, and can lead to lower response rates because they take longer to complete (Dillman et al., 1993).

Dillman, Tortora, Conradt, and Bowerk (1998a) conducted a study to compare the response rate of a plain web-based survey (without the use of colors or graphics) to an advanced design web-based survey which incorporated a number of features to make the experience easier for participants. These features included bright colors and graphics on every page, a constantly changing figure/ground format to help participants quickly visually align survey items with answer choices, HTML tables for the setup of the questions, and placement of answer categories to the extreme right near the scrolling bar to minimize movement of the cursor between answer selections and scrolling of the page. The results indicated that the plain web-based survey produced higher response rates than the advanced design web-based survey. It is important to note that the advanced design web-based survey took more than 3 times as long as the plain web-based survey to transmit the survey and display all content.

Useful guidelines and principles for the implementation and design of web-based surveys have emerged (Dillman, 2000; Dillman et al., 1998b; Schonlau et al., 2002), although others recommend the use of plain and simple web-based surveys arguing that principles of good paper survey design also apply to web-based surveys (Ranchhod & Zhou, 2001; Peat, 2001). Little is known about the effects of web-based survey design on response rates or response accuracy; further research is warranted in this area.

The Study

In the spring of 2003, a study was conducted in the College of Education at a large urban research institution in the Southeastern United States. This exploratory study was designed to test a new instrument measuring web-based instruction using a WebCT survey. The institution had committed to using WebCT to deliver instruction, content and materials. All instructors of courses in the College of Education that were using WebCT for their course were asked to participate in the study. Sixteen courses (312 students) were included in this study. A total of 184 students completed and submitted a survey yielding a 59% response rate.

For this study, a paper survey of 20 multiple-choice items was created and then replicated electronically. Following the findings of Ranchhod and Zhou (2001) and Peat (2001), no additional technology was used to enhance or modify the web-based survey with multimedia capabilities. The survey was created offline and then implemented online via the WebCT survey tool. Creating a survey in WebCT was tedious. Survey questions must be individually placed in the question database within WebCT. Once in the database, questions are individually selected to be added to a newly created WebCT survey. Unfortunately, the survey tool in WebCT did not permit copying, moving or downloading a survey intact. This process would have to be repeated for each of the sixteen courses. Fortunately, other programs are available to overcome this time-consuming method of creating, copying and moving surveys into multiple WebCT courses. Respondus (www.respondus.com) is a third-party software program that enables the user to construct complete surveys offline and outside of the WebCT environment. Respondus permits the user to easily copy, move, upload and download surveys into WebCT intact. This program was used to construct and transfer the survey into the 16 WebCT courses included in the study.

Findings and Discussion

One of the drawbacks of using the WebCT survey is the variation in the survey setup in different versions of WebCT. Several of the 16 courses in this study were offered throughout the State University System, while others were offered only at the institution of the study. The State University System used WebCT version 3.1; the institution used WebCT version 3.6. Due to these version differences, the survey setup required different accommodations on each version.

One of the strengths of WebCT is the flexible nature in which course content can be accessed and delivered non-linearly. Ironically, this strength translates to a weakness when dealing with the survey tool. The linear process of the survey: the consent form, instructions, then the survey, needed to be strictly maintained. Modifications needed to be made to every WebCT course to prevent access to the survey from multiple points and to reconfigure it so the consent form, instructions and survey appeared in strict sequence. Survey access had to be removed from the students' personalized "myWebCT" area and their course menu. The survey needed to be delivered on a separate path from the other non-study surveys to allow this linear process to be facilitated. There were also additional procedures that were employed to appropriately set them up in each of the sixteen WebCT courses. This entire process of configuring a survey within WebCT was quite tedious. After it was uploaded into a WebCT course, it took the researcher 113 steps/"clicks" per course to properly stage the survey for students.

An initial testing of the survey access on campus computers and from a home computer with several versions of the web browsers Internet Explorer (IE) and Netscape Navigator (NN) proved successful. Unfortunately, some students attempting to access these surveys from off-site locations used other versions of IE and/or NN. Although they were able to access e-mail and other WebCT tools in their course, many experienced technical problems with the survey tool that prohibited them from accessing and/or completing the survey. (Some WebCT tools (survey and chat tool) or unique configurations in WebCT have been known to be very browser and version specific. Recommendations were made to participants to use institution computers or particular browsers/versions but this was not feasible in many cases.) After several participants had reported these problems, each WebCT course was modified with enough flexibility to eliminate the previous problems. All participants were contacted by WebCT e-mail and asked to retake the survey if they had been unable to previously access the survey.

WebCT automatically tracks the number of individuals that begins and subsequently completes a survey. Records showed that 198 students attempted the survey but only 184 completed the survey. Although it would be useful to know the reasons why 13 students did not complete the surveys, it was impossible to determine if this was due to technological problems or for other reasons. These is some anecdotal evidence that suggests that participants of web-based surveys tend to have shorter attention spans and therefore surf from the survey to other web pages prior to survey completion (Fricker & Schonlau, 2002). In addition to the time it took to setup, maintain and download the data from the survey in each course, a certain level of skill was required in these processes along with periodic checks of all the courses on two different versions of WebCT, as well as the regular tweaking of the survey due to unseen problems. Furthermore, following the completion of the survey, each WebCT course needed to be restored to its original state; all evidence of the survey and the accompanying files had to be removed from the course. The combination of all these tasks amounted to substantial work within the courses on a daily basis over several weeks for the researcher.

Having direct access to the WebCT courses as an "administrator" provided the researcher with opportunities to address any problems and issues quickly. Participating students e-mailed the researcher or phoned him with questions, concerns or problems regarding the survey. Ready access also allowed the researcher to regularly track students' response rates throughout the survey period. This information was then provided to faculty members who would encourage and remind their students to complete the survey. Several problems occurred when downloading the data survey from the WebCT courses. Similar to the issue some participants had in accessing the survey, these problems related to using different web browsers and versions. Due to a bug in Internet Explorer 5.5 Service Pack 1, it was impossible to download survey data with this browser version. Instead Netscape Navigator 4.79 was used to download the survey data. (Newer versions of Netscape Navigator and Internet Explorer were incompatible with WebCT at the time). Ironically, study participants experienced fewer problems when they accessed and completed the survey with Internet Explorer 5.5--a number reported problems completing the survey using Netscape Navigator 4.79.

As described earlier, different versions of WebCT required different setups for the survey. Different versions of WebCT also generated different data files. One version of WebCT (3.1) produced data files with extraneous code: trailing HTML tags accompanied the survey data file of these courses. These files needed to be "cleaned up" and stripped of this additional coding before further analysis could continue. The newer version of WebCT (3.6) generated accurate data files void of this trailing code. An additional problem arose in the transfer of data from WebCT to another software program, the Statistical Package for the Social Sciences (SPSS). This was due to differing file name sizes in WebCT and SPSS. In WebCT each question title had a limit of fifteen characters. These question titles were automatically converted to variable names in SPSS. The researcher used the first 10 characters of the question titles to identify the general category and the last 5 characters to identify the specific variable. In contrast, SPPS limited variable names to only eight characters. As a result of examining the first eight characters of the question titles, SPSS found many duplicates and outputted an error message, "The variable name (truncated to eight bytes) conflicts with a previously defined variable". To correct this problem, once the data were uploaded into SPSS all variables were renamed with shorter, but appropriate and descriptive names that were in compliance with the eight-character variable name limitation in SPSS. In summary, the problems encountered and suggested solutions to using the web-based WebCT survey in this study were:

Problem: Variations in processes required with survey setup and downloading data due to using different versions of WebCT. Suggestion: If at all possible, try to limit survey use to one version of WebCT.

Problem: Only some web browsers and certain versions of these browsers were compatible with completion of the survey and downloading of the data. Suggestion: Learn what the most stable browser and version is for the survey tool and attempt to standardize on it.

Problem: The length of data variables in WebCT and SPSS differ. Suggestion: Use the lowest common denominator--SPSS variables were limited to 8 characters (and WebCT 15). When naming WebCT variables, limit them to 8 characters.

Regardless of the CMS, a number of general obstacles with web-based surveys can be anticipated and many overcome, if not avoided entirely. These include browser incompatibility and technical issues related to software/hardware problems. As new browser versions are released, it takes time for CMSs to "catch up". Using older tested and compatible browsers can avoid the potential compatibility problems with new browser versions. Because of technical problems related to hardware/soft-ware

issues, providing the survey in other formats (e.g. paper-based survey, telephone interviews) maybe critical to maximizing the response rate (Dillman, 2002; Schonlau et al., 2002).

Conclusion

Web-based surveys are becoming a popular format for survey research. As the demand for web-based surveys continues to grow, it becomes imperative to better understand the issues and obstacles that are encountered in using surveys within CMSs. The automation in implementing web-based surveys in CMSs, contacting participants and collecting the data make this an attractive venture. But as this study showed, these benefits are offset by the drawbacks of web-based surveys including the incompatibility with different versions of the CMS, the incompatibility with different browsers and versions of those browsers, and the tremendous time needed for all stages of the web-based survey process. As CMSs continue to mature and reach new levels of sophistication, it is expected that many of these limitations will disappear and new ones emerge. Clearly, more and continued research is needed to examine the challenges and opportunities that arise in implementing web-based surveys in these environments.

References

Dillman, D. A. (2000). Mail and interact surveys, The tailored design method (2nd ed.), New York: John Wiley & Sons.

Dillman, D. A., R. D. Tortora, J. Conradt, & Bowerk, D. (1998a). Influence of Plain vs. Fancy Design on Response Rates for Web Surveys, unpublished paper presented at the Annual Meeting of the American Statistical Association, Dallas, TX.

Dillman, D. A., R. D. Tortora, J., & Bowerk, D, (1998b). Principles for Constructing Web Surveys. SESRC Technical Report 98-50, Pullman, WA.

Dillman, D. A., Sinclair, M.D., & Clark, J.R. (1993). Effects of questionnaire length, respondent-friendly design, and a difficult question on response rates for occupant-addressed census mail surveys." Public Opinion Quarterly, 57, 289-304.

Fricker, R.D., & Schonlau, M. (2002). Advantages and disadvantages of internet survey research surveys: Evidence from the literature. Field Methods, 14(4), 347-367.

Nichols, E., & Sedivi, B. (1998). Economic Data Collection Via the Web: A Census Bureau Case Study. 1998 Proceedings of the Section on Survey Research Methods. American Statistical Association, Alexandria, VA, 366-371.

Peat, J.K. (Ed.). (2001). Health science research: A handbook of quantitative methods. Sydney: Allen and Unwin.

Ranchhod, A., & Zhou, F. (2000). Comparing respondents of e-mail and mail surveys: Understanding the implications of technology. Marketing Intelligence and Planning, 19(4), 254-262.

Schonlau, M., Fricker, R., & Elliott, M. (2002). Conducting research surveys via e-mail and the web. Santa Monica, CA: RAND.

Marwin Britto, Central Washington University, WA

Dr. Britto is an assistant professor and Director of the Educational Technology Center.