Development of a Web-based student satisfaction survey. (The scholarship of teaching and learning).

Abstract

There is an increasing emphasis in higher education on assessment of student learning. Some universities incorporate the business model of measuring customer satisfaction into their assessment of academic programs and student support services. There is a need for comprehensive assessment in distance education programs, particularly that of the growing field of online education. This report describes a Web-based student satisfaction survey about the services provided by a distance graduate education university. Rather than report the findings of the survey, this report describes the development of the Web-based questionnaire, the administration of the survey on the World Wide Web (WWW), how the results were used, the limitations of the survey and recommendations for refinement, and future use of Web-based student satisfaction surveys.

**********

In higher education the assessment of student learning includes measurement of processes that influence learning, services that contribute to student learning, and student satisfaction with services (American Association for Higher Education, 2001). Student satisfaction with non-instructional services such as the office of financial aid and tutoring may be included (Non-Instructional outcome assessment plans, 2001).

For-profit businesses emphasize measurement of customer satisfaction in order to determine the effectiveness of services, identify areas needing improvement, and for use in the design of business plans (Dickey, 1998; Drake, Gwynne & Waite, 1998). The parameters of customer satisfaction (McColl-Kennedy & Schneider, 2000) and research measures (Nicholls, Gilbert & Roslow, 1998) have been defined. While the World Wide Web (WWW) has become an important mechanism for universities to collect and report assessment data (Tyree, 1997), the emphasis in online distance education tends toward the evaluation of courses, such as efforts described by Robinson (2000), rather than the broadly defined areas of assessment.

The purpose of this report is to describe a web-based survey of student satisfaction at a distance graduate education university. However, rather than present data on student satisfaction, this report describes the development and administration of the survey, the limitations of the approach, and suggestions for improvement so that universities that consider Web-based student surveys may understand the requirements and limitations of such an undertaking.

Methods

Setting

The university at which this study was conducted is a for-profit corporation that has been providing graduate level distance education for over 30 years and is regionally accredited by a regional accrediting organization. At the time of the survey the university offered PhD degree in five areas and master's degrees in two areas. Some degrees are offered entirely through structured online courses, others through a combination of a mentored and course-based curriculum; some require short residencies. The University specifies minimum computer hardware, browser and software requirements and each student is provided with a unique University electronic username and password for e-mail and access to restricted University Web pages.

University administrators decided that a comprehensive customer satisfaction survey of all students should be carried out to assess students' utilization of, and satisfaction with, each of the academic programs and academic support services the University provides to its students. Because of the University's reliance on the Internet and the WWW, a decision was made to conduct a Web-based survey rather than a mail or telephone survey. A program director was appointed to administer the survey. The University's Institutional Review Board (IRB) for the protection of human subjects approved the survey as a research study.

Development of the Survey

Since the University had not conducted a comprehensive survey of students, both the questionnaire and the survey administration method had to be created. The variety and uniqueness of the degree programs' curricula required that a tailored questionnaire be developed. Inclusion of the results in an upcoming accreditation self-study report required that the survey be completed within four months.

A list of possible survey domains was developed for review by University administrators. Following their approval draft questions were developed for their review. The supervisors of each of the University's student support service areas (e.g., Academic Counseling, Residencies, Library Services, etc.) were also provided with the draft questions, related to their respective service area, for review and correction.

The survey included 236 questions covering all aspects of the academic programs (faculty, program directors, residencies, academic affairs office, etc.) and student support service areas (e.g., academic counseling, enrollment counseling, library, financial affairs, etc.). Questions were also included about prior academic preparation, initial impact of the program on their life, their opinion of the quality of the program versus the monetary cost, and demographic characteristics. Because student evaluation of online courses is included at the end of each course, the survey did not include questions about program courses.

Some questions were unique to each service area but a similar "core" set of structured questions inquired about students' satisfaction with each service area. This "core" set of questions assessed: a) how much the student had used each service, b) the timeliness of the response from the service area, c) students' satisfaction with the response time, d) students' perception of the courtesy of the persons with whom they interacted, e) students' perception of the helpfulness of the interaction, f) students' reasons for interaction with the respective service area, and g) the student's usual mode of communication with the respective service area. Student satisfaction with the response time was important because University policy specifies the maximum time for some services to respond to students.

Each question in the questionnaire could to be made conditional upon the responses to a single question or multiple questions so that only students with specific experiences (e.g., had used a specific University service) or characteristics (e.g., a doctoral student) would be presented with subsequent questions on a given topic. Because the survey included such "item branching" or conditional questions every student was not asked to respond to all 236 questions in the survey.

The University's Instruction Technology (IT) staff determined that because of the time limitation and the complex nature of the questionnaire that it would be more efficient to create a program with an accessibility and verification system than to review, purchase, and install a commercially available web-based survey program and then develop the questionnaire. Hypertext Markup Language (HTML) and Cold Fusion were used to develop a program, resident on the University's servers and available on its Website, to create questions, administer Web-based surveys and enter the responses into an Access database.

The survey program permitted inclusion of structured or open-ended responses, or both, to any question. Structure questions could be multiple choice, true/false, yes/no, or priority ranking, with at least 16 possible response choices per question. Narrative responses could be limited to either 50 or 100 words. Thirty-five 35 "item branching" questions were included in the questionnaire. University administrators and a group of students were given an opportunity to test the survey as it would appear "live" on the University's web-site and to make suggestions for revisions. This preview indicated the survey took from about 20-60 minutes to complete.

Administration of the Survey

The University's Records Office provided a list of University e-mail usernames and passwords of the 1,591 students enrolled in the University at the beginning of the quarter during which the survey was administered. The IT staff developed a program that connected the list of current students' usernames and passwords to a web-based security page that would permit only currently enrolled students to participate in the survey and that would permit entrance into the survey only once.

An invitation to participate in the survey was sent in a brief e-mail message from the Vice President for Academic Affairs to an electronic mailing list for all University students. The e-mail included a link to the Uniform Resource Locator (URL) for the survey's security page and a word-processed attachment containing the IRB approved consent statement. The invitation emphasized the need for student feedback in order to ensure high quality of educational services, and included the dates and time the survey would be accessible (the fourth week of January). Mid-week during the survey each program director sent an e-mail on their respective program's student electronic mailing list encouraging students to participate. Two days before the survey was closed, the student government representative sent a message to all students encouraging them to participate. The survey administrator and Help Desk staff were available via e-mail and telephone to assist students who had technical difficulties.

The first Web page of the survey's security system informed respondents of the estimated time to complete the survey and that they would have only one opportunity to participate, and gave them assurance that the security pages were separate from the survey itself in order to help provide anonymity. A copy of the IRB approved consent statement was included. To proceed, respondents were required to click on an "I Agree" button acknowledging that they had read the consent statement and agreed to participate. They were also given the option of exiting. Once the respondent entered her/his username and password the program took them to a web page that used a Common Gateway Interface (CGI) script to verify that the respondent was eligible to participate and had not previously taken the survey. Those eligible were presented with a URL link to the first page of the survey. The first page included instructions about how to answer questions, and informed respondents that if they used the back button of their browser to go back through the survey they would have to repeat their answers to all subsequent questions. They were informed that demographic questions were optional and would be used to compare the survey respondents with the student population.

The program presented one question per Web page and respondents used their mouse to click on the available responses or type a narrative response in a blank box before clicking on a "continue" button to proceed to the next page. Respondents received a message informing them of their progress through the survey at the completion of each one-quarter of the questions. After the last question, respondents were thanked for participating and they were asked to click the "submit" button, which submitted all of their responses into the survey database.

At the end of the week a message was sent on the student electronic mailing list announcing that the survey was closed and thanking participants. One month after the close of the survey, at the request of the student government representative, an announcement was sent to all students on the student electronic mailing list that the survey was reopened for an additional week. The University's IT staff set up the survey security pages so that only those students who had not participated during the first week of the survey would be admitted during the second week.

Use of the Survey Results

The Vice President of Academic Affairs sent reports of both the survey's quantitative results and narrative responses as e-mail attachments to program administrators. Supervisors of each student support service area were sent the survey results specific to their area. Supervisors were asked to develop a plan for addressing any findings that indicated a needed improvement, and to propose a plan for continued collection of student satisfaction data. Summary reports of the results were sent as e-mail attachments to the electronic mailing lists of all faculty members and all students. A hardcopy summary report of the survey results was included in the library of materials made available to the regional accreditation site visit team. Written and oral reports about the survey were presented to a committee of the University corporate Board. An oral report about the results was made to a group of faculty and administrators participating in a residency session. Highlights of the results were published in the University's quarterly newsletter sent to administrators, faculty, students, alumni and University Board.

Results

The results presented in this report are limited to results from the first one-week survey period and to experiences with the technology related to administration of the survey. Based upon the largest number of students answering any single question on the survey (the demographic information question), 18% (282) of eligible students participated in the survey; 114 accessed the survey within the first 40 hours (during the second one-week survey period 67 students accessed the survey). A comparison of selected characteristics of survey respondents with characteristics of the University's student body (Beere, 2001) shows that the respondents, while differing from the student body, did, to a degree somewhat reflect the student population on mean years of age (44 years in both groups), sex (66% female vs. 58% female), race (21% African American and 77% White vs. 26% African American and 58% White), respectively.

During the week the survey was administered 46 students contacted the survey administrator for assistance with technical problems they had experienced while trying to take the survey. Thirty-seven percent (17) of those students had completed or nearly completed the survey when they were either "kicked out" of the survey or received an error message when they tried to submit their responses. The Help Desk staff reset the security system to readmit those students who had been "kicked out" of the survey and who requested the opportunity to attempt the survey again. Another 13% could not access the survey because they did not know their username and password, or the username and password they used was not accepted by the security system. In cases in which students had problems with username or password the Help Desk staff verified the usernames and passwords and eligibility, and informed the student. Because some students could not open the attached consent statement, the invitation to participate in the survey was resent to all students with the consent statement embedded in the text of the e-mail. On the first day of the survey IT staff corrected an error in the survey program so that narrative responses that included apostrophes, parenthesis and quotation marks would not result in students receiving an error message when they submitted their responses. Fewer than six students sent e-mail letters to the administrator stating that they would not participate in the survey or reattempt the survey (if they had been "kicked out") due to its length. After completing the survey several students wrote e-mail letters to the administrator that were complementary of the survey and thanked the University for seeking student involvement through the survey.

Discussion and Recommendations

The student satisfaction survey questionnaire and Web-based survey administration program provide the foundation for future development of student surveys at the University. Refinements could lead to greater efficiency and higher student participation. Additional research should be conducted with a variety of types and levels of incentives to try to increase student response rates. The University should also consider periodically administering sections of the questionnaire about specific services to targeted groups of students rather than administering the entire survey to all students. With sufficient refinement, and incorporation into institutional research functions, the student satisfaction survey, described here could become an important part of the University's comprehensive plan for assessment of student learning. Before investing additional resources into the refinement of the Web-based survey program described here, the University should determine whether or not there are commercially available web survey programs that would serve the University's purposes in future surveys.

The survey results were limited by several factors. The response rate was lower than generally found in Web-based surveys (Cook, Health & Thompson, 2000) and thus generalizability of the results about student satisfaction is limited since the respondents may not be reflective of the entire student population. The survey results were also limited because reliability and validity testing of the questionnaire, other than "face validity," was not carried out.

The invitations to participate in the survey may have deterred some students from participating since they included estimates of the length of time to complete the survey (Crawford, Couper & Lamias, 2001). After the survey reported here a different questionnaire of 10 structured questions embedded in the body of an e-mail with a Header message stating that it was a 3-5 minute survey was sent to 300 students in two of the doctoral programs. This survey also resulted in a low response rate (12%) (Wiist & Graham, 2001). In contrast to the Web-based survey and the e-mail survey, 88% of 233 students who completed an online course during the previous year completed the web-based course evaluation questionnaire at the end of the course (Wiist, Graham & Brown, 2001).

Some research suggests that the progress messages in the questionnaire may have deterred completion of the survey (Crawford, Couper & Lamias, 2001). In addition, the low participation rate might have been due to students' lack of trust in the assurances of confidentiality and anonymity since many Internet users are aware of the technological capacity to track Web users (Simsek & Veiga, 2001). Although the questions were likely salient, the lack of other incentives may have contributed to the low response rate (Stanton & Rogelberg, 2001). In contrast, the technical difficulties reported by some survey participants suggest that administrative assumptions that students had a high level mastery of the Internet and WWW may have been incorrect.

Technological factors may also have prevented some students from participating in the survey. The Help Desk staff concluded that the web browser versions or settings and computer equipment of some students resulted in difficulty completing the survey. However, many students were unable to complete the survey or submit their responses because their Internet Service Provider (ISP) disconnected them because it interpreted time spent in the survey as being "inactive." Including several questions per page and/or allowing suspension and reentry into the survey might alleviate this problem.

Analysis and reporting of the survey results was hindered because the data was entered uncoded, in alpha form (words) into an Access database. This required recoding of some data into Excel and subsequent conversion into an SPSS database. Clear mutual understanding between programmers and survey developers about the structure of the survey database is needed to prevent such problems.

In many respects the published recommendations and guidelines for the development and administration of customer and student satisfaction surveys, including Web-based surveys, were adapted in the student satisfaction survey reported here. Despite the limitations of the survey resulting from a low response rate and certain the technological features of the survey, the University now has a foundation on which to develop a comprehensive assessment of student learning. In addition, some of the survey questions could be used to assess staff, faculty and alumni satisfaction with selected University services. Other Universities may use this report as a guide in the development and administration of Web-based student satisfaction surveys.

Reference List

American Association for Higher Education Assessment Forum Definitions of Assessment. (n.d.). Retrieved October 26, 2001 from http://www.aahe.org/assessment/assessfaq.htm# Definitions of Assessment

Beere, C. (2001). Walden, Fall 2001. Unpublished presentation. Minneapolis, MN: Walden University.

Cook, C., Heatlh, F., & Thompson, R.L. (2000). A meta-analysis of response rates in Web- or Internet-based surveys. [Electronic version]. Educational and Psychological Measurement, 60, 821-836.

Crawford, S.D., Couper, M.P., & Lamias, M.J. (2001). Webs surveys: perceptions of burden. [Electronic version]. Social Science Computer Review, 19, 146-162.

Dickey, J.D. (1998). Creating a customer satisfaction measurement system. [Electronic version]. Industrial Management, 40, 8-11.

Drake, C., Gwynne, A., & Waite, N. (1998). Barclays Life customer satisfaction and loyalty tracking survey: A demonstration of customer loyalty research in practice. [Electronic version]. International Journal of Bank Marketing, 16, 287-292.

Kay, B., & Johnson, T.J. (1999). Research methodology: Taming the cyber frontier: Techniques for improving online surveys. [electronic version]. Social Science Computer Review, 17, 323-337.

McColl-Kennedy, J., & Schneider, U. (2000). Measuring customer satisfaction: Why, what and how. [Electronic version]. Total Quality Management, 11, S882-96.

Nicholls, J.A.F., Gilbert, G.R., & Roslow, S. (1998). Parsimonious measurement of customer satisfaction with personal service and the service setting. [Electronic version]. Journal of Consumer Marketing, 15, 239-253.

Non-Instructional outcome assessment plans: Financial aid. (n.d.). Retrieved October 26, 2001 from Eastern New Mexico University Assessment Resource Office Web site:http://www.enmu.edu/users/smithl/Assess/noap /1999_2000/plans/Financialaid.htm

Non-Instructional outcome assessment plans: Learning support. (n.d.). Retrieved October 26, 2001 from Eastern New Mexico University Assessment Resource Office Web site: http://www.enmu.edu/users/smithl/Assess/noap/1999_2000 /plans/LearningSupport.htm

Robson, J. (2000). Evalulating on-line teaching. [Electronic version]. Open Learning 15 (2), 151-172.

Simsek, Z., & Veiga, J.F. (2001). A primer on Interact organizational surveys. Organizational Research Methods, 4, 218-235.

Stanton, J.M., & Rogelberg, S.G. (2001). Using Internet/Intranet Weg pages to collect organizational research data. [Electronic version]. Organizational Research Methods, 4, 200-217.

Tyree, R. (1997, October). Assessing with the Net: Using technology to know more about students. Retrieved October 26, 2001 from http://www.aahe.org/bulloct.htm.

Wiist, W.H. & Graham, M. (2001). [Health and human services students assessments of first year faculty advisors and faculty mentors]. Unpublished raw data.

Wiist, W.H., Graham, M. & Brown, R. (2001). [Summary of student evaluations of Social and Behavioral Science Foundations 8417]. Unpublished raw data.

Author Note

The author thanks Mark Schmidt for developing the Web-survey program, Waris Hydrie for developing the Web-survey security system, and the students who participated in the survey.

William H. Wiist, Walden University, MN

Dr. Wiist is the Director of the PhD Program in Health Services. He has directed surveys of more than 10,000 individuals.