E-mail versus Web survey response rates among health education professionals.

Abstract: To determine whether there is a difference in response rate from a survey given via e-mail or the Web, 600 professional health educators were divided into two groups. Three hundred received an e-mail survey and 300 were sent an e-mail asking them to go to a Web page to complete a survey. No statistically significant differences were found in the response rate between the two groups. The Web survey had a 48% response rate whereas the e-mail survey had a 43% response rate. The Web and e-mail can yield similar results when collecting data, and both may have unique advantages.

**********

The phenomenal growth of Internet technology use in recent years has prompted many health education professionals to find ways to use this technology to communicate more effectively. A variety of Internet lists and discussion groups now exist for health education professionals to have wider and more rapid discussions of current topics. Chat room capabilities are also available for even quicker communication. The Web has grown into a massive organism that contains both quality and questionable health-related information. In addition, more health educators are using the Internet to facilitate needs assessment, to investigate strategies for program development, or to collect data in their research (Bernhardt & Hubley, 2001). However, because the Internet (which includes e-mail and Web-based activities) has some protocols and use patterns that differ from other survey methods, many health educators who use survey methodology in their research are unprepared to effectively use this technology as a data collection tool (Cook, Heath, & Thompson, 2000).

Although e-mail has been around for more than 25 years, it has only become popular among larger segments of the population within the last decade (NTIA and the Economics and Statistics Administration, 2002). The Health Education profession experienced the first initiatives for organized e-mail with the HEDIR directory in 1992, and the development of the HEDIR list (HEDIR) in 1994 (Kittleson, 1997a). Since then, numerous lists have been created and used for a variety of health education efforts. Because the profession believes e-mail has advantages over paper mail, it has reasoned that data collection via Web-based surveys would also have advantages. For example, Tse (1998) asserts that e-mail surveys are cheaper, faster, less likely to be ignored, and more environmentally friendly; implying that individuals should be more likely to respond.

In the mid 1990s several studies (although few in health education) tried to determine how effective e-mail was for collecting data. E-mail was initially compared to postal rates on a variety of variables including cost and response rates (Sheehan, 2001). About this same time, the Internet became friendlier to use, and developing Web pages became easier. Consequently, researchers also began using the Web as a data collection tool.

Initial studies found returns via postal service were higher than e-mail returns (Anderson & Gansneder, 1995; Kittleson, 1995; Mavis & Brocato, 1998). However, some early studies among businesses with extensive e-mail use (especially e-mail within a network system) found e-mail response rates were significantly higher than postal service rates (Parker, 1992; Sproull, 1986). It should be noted that most of the surveys with high e-mail response rates were done prior to 1995 when e-mail was still relatively novel. It is hypothesized that rates have since declined as the novelty has worn off (Sheehan, 2001). In the present age of cheap, easy-to-obtain, spam-polluted (unwanted messages) e-mail, it may be more difficult to obtain high response rates.

The Web presents a new opportunity in data collection. Unlike e-mail, a Web page can be formatted to allow for anonymous response (Kittleson, 1995). In addition, Web format can offer more attractive pages, present a more authoritative appearance--such as logos, and be easier to read than e-mail (Dillman, 2000). However, few studies have measured the effectiveness of e-mail for health education survey research, and even fewer use of Web pages. It is important to determine whether either of these methods can be a viable mode for data collection and how they compare. The purpose of this study was to determine if there is a difference in the response rate between an e-mail survey and an e-mail-recruited, Web-based survey among health education professionals.

METHODS

PARTICIPANTS

Participants were selected from the HEDIR directory. The HEDIR directory (www.hedir.org) lists health educators from a variety of worksite settings who agree to participate in the HEDIR list. At the time of selection, 984 individuals were listed in the directory. Only university-based subscribers (N = 600) were selected to participate in the study.

An individual cannot be on the HEDIR unless he or she has an e-mail address and access to the Internet to register for the list. Therefore, it is possible that participants on the HEDIR do not reflect the general population of health education professionals. However, the HEDIR has one of the most comprehensive lists of e-mail addresses for health educators throughout the world. Thus, realizing the limitations, it was considered the most practical population for this study.

Following approval by the researchers' institutional review board, the 600 participants were assigned to one of two groups using a random-number procedure (Babbie, 2000). In one group, e-mails were sent to all individuals asking them to complete a short survey via an e-mail reply. Participants were asked to click "reply" to the e-mail and to asterisk their responses on the survey. They were also instructed that if the 'reply' response did not contain the original 'survey', they were to copy and paste the survey from the original message to their 'new' message. Participants were informed that their responses would not be anonymous, but that the researchers promised to remove all identifying information from the e-mail.

The second group was also sent an e-mail, but instead of being asked to complete the survey via e-mail, they were given a Web address at which to complete the survey. Web participants were assured that their responses would be anonymous. Participants were asked to either click the link in the e-mail or cut and paste the URL into their preferred browser. Once at the appropriate Web page, they were asked to complete the survey on the Web.

A drawback with using the Web is the possibility that some may complete the survey more than once. This possibility was reduced by having the Web survey record the Internet protocol (IP) addresses including the date and time along with the respondent's answers. The IP address is a special code (series of numbers) that every computer connected to the Internet possesses. Those targeted for the sample were all professionals and likely were responding from computers with static (permanent) IP addresses. The IP address was used by the researchers merely to determine whether any responses came from identical computers. In those instances, the second, and all subsequent responses, from that particular IP address were eliminated from the results.

INSTRUMENTATION

It should be noted that the questionnaire used in this survey is of minimal interest since the purpose of the study was only to determine which survey strategy would obtain the highest response rate. However, to prevent the respondents from knowing the true purpose of the study, the researchers used a survey that had been tested previously. Eight health educators, who are experienced e-mail and Web users, reviewed the survey for wording and ease of completion. Their reviews indicated the survey to be quick, easy, and understandable. In previous studies, the survey was completed in approximately three minutes (Kittleson, 1995).

The instrument consisted of six items, including one demographic question. Participants were asked their professional status (i.e., full-time employed, part-time employed, full-time student, part-time student). They were then asked about their subscription practice for five popular health education journals. For each of these journals, they were asked to select one of three possible subscription practices: a) they "currently" subscribe to/receive the journal; b) they have "ever" subscribed to/received the journal (assuming they did not currently receive it); and c) they have "never" subscribed to/received the journal. Participants responding via e-mail were asked to place an asterisk in front of their desired response. Those who responded via the Web, had the opportunity to 'point and click' one of three associated buttons to indicate their choice.

DATA COLLECTION

On Day 1 of the study (Tuesday), all six hundred participants were sent their respective e-mails. Since it is common for an e-mail server to be temporarily down, the researchers' mail system automatically re-sent the e-mail message four hours after the initial attempt. If a second undeliverable notice was received, it was noted and the participant's e-mail was removed from the nickname list.

Following Kittleson's recommendation (1997b), reminder notices were sent out on Day 4 and Day 8. For those in the e-mail group, the message was sent only to those who had not yet responded. The message was identical to the first mailing with the addition of a statement indicating that we did not receive a response to our earlier request. The survey was also included with the reminder.

Since the researchers could not determine which subjects in the Web group had responded, all participants in this group were sent reminders. The message began by apologizing to those who had already responded, but then asked those who had not responded to please complete the survey.

Four days after the second mailing, a third e-mail was dispatched to each group encouraging participants to complete their respective surveys. The researchers continued to count all responses for an additional four days. Any responses sent after that time were not included in the data processing.

RESULTS

Of 300 attempts, 291 individuals received the email survey and 129 returned the survey. Figure 1 shows the days that responses were returned. This reveals an e-mail response rate of 43.0% of all attempted recruits. There appeared to be no duplications among this group (based on no identical e-mail addresses).

Of 300 attempts, 294 individuals received the Web-based survey. One-hundred fifty-three surveys were completed on the Web. Ten messages had IP addresses the matched earlier messages. All of these ten duplicate messages were sent within seconds of the first response from the same IP address, which seems to indicate that these participants inadvertently clicked the "send" button twice. The 10 duplicate responses were deleted, making the final total 143. This represents a Web-based response rate of 47.7% of all attempted recruits. A chi-square analysis of response rates found no statistically significant difference between the two groups ([P.sup.2] (1, n = 272) = 0.40, p = 0.53).

If results are analyzed by day of the study, some minor findings are worth noting (see Figure 1). Nearly 50% of the total responses from the Web group took place the same day the first message was sent. The e-mail group also had a high percentage return the first day (42.6%). This supports Yun and Trumbo's (2000) study that showed the greatest response for e-mail or Web methods occurring in the first 72 hours. Within two days after the first reminder (on Day 4), nearly twice as many e-mail responses were returned than Web responses. The day of the second reminder (Day 8), e-mail responses were also high compared to Web responses. The following day (Day 9) more Web responses were received than e-mail responses.

DISCUSSION

Interpretation of these results must take into consideration the participants chosen for this study. The sample was from a population of health educators who voluntarily participate in a computer-based discussion group (the HEDIR). Therefore, all of the respondents have access to e-mail and the Web either at home or work. In addition, because the HEDIR serves as an invaluable tool for sharing information, collecting ideas, or discussing current events, these participants may have a favorable opinion of computer-assisted communication (Kittleson, 1997a).

Based on the results of this study, two conclusions may be drawn: a) for health educators represented on the HEDIR, the Web has no response rate advantage over collecting data via e-mail; b) as evidenced by the relatively moderate response rates, the health professionals involved with the HEDIR are not very responsive to either type of Internet survey despite their experience using these technologies to share information.

For health educators who are excited about using the Web to collect data, the results, on the surface, appear to be somewhat disappointing: no statistically significant difference between the two groups. In addition, it should be noted that the response rates were low relative to other methods (see Table 2). They were also lower than earlier studies using e-mail follow-ups (Cook, Heath, & Thompson, 2000; Kittleson, 1997b), which had response rates between 50 and 60%.

The lure of the Web, with online forms and respondent anonymity, apparently was not enough for the majority of the health educators in this study to complete the survey online. The similar response rates for the e-mail and Web versions of the survey may mean that the one extra step, from the e-mail message to the Web, was enough to prevent many of the Web recruits from participating. This may be because many users are accustomed to e-mail systems that allow individuals simply to dick a link to be automatically transferred to a particular Web page. An alternative explanation is that a brief, easy-to-complete e-mail survey prompted a similar number of e-mail responses. In addition, since the nature of the questionnaire was impersonal and non-threatening, many e-mail respondents may have felt no apprehension to completing the survey even though it was not anonymous. It can also be concluded that the similar response rates mean that health education researchers wishing to conduct research now have the option of using either the Web or e-mail.

According to O'Rourke (1999), response rates can be affected by any of the following: a) identification with a known entity; b) informing the respondent; c) making it easy; d) keeping length reasonable and considering respondents' lack of time; e) respondent fear or unusual personal inquiries; f) timing; g) doing follow-ups; h) respondent interest, and i) offering an incentive. Each of these factors is considered below.

For consistency, the researchers did not use "official" stationary or logos for identification in the e-mails or Web page. However, the study was identified with a popular entity (the HEDIR) and the researchers were affiliated with a well-known university. One cannot be certain, but on balance, it appears that this concept would not account for the low response rates.

With regard to informing the respondents, it should be noted that this study, like all surveys coming from the researchers' institution, was required by the Institutional Review Board to state clearly the purpose of the study. The IRB also understands the importance of, and permits, misleading the respondent from the real purpose of the study--assessing response rates. Again, it is doubtful that this was a factor in the relatively low response rates.

Making the survey easy, relatively short, and considering the respondent's lack of time are also factors that can affect response rates. All of these were also addressed in this study. The survey was relatively easy to answer and took less than three minutes to complete. In fact, the quickness of the survey may help explain the fact that nearly half of all respondents in both groups responded on the Day 1.

Other factors that can affect response rates include respondent fear, unusually personal inquiries, doing follow-ups, or the timing of the study (O'Rourke, 1999). Fear should not have been an issue since all participants were assured of confidentiality. Some may have felt uncomfortable that their results could be tracked to them. However, it should be noted that the questions asked in the survey were non-threatening (i.e., what professional journals they had received). Fear of using technology also should not have been an issue since all of the subjects were selected from a very active listserve that requires information be provided via the Web. As discussed, both groups received two sets of follow-ups even though most of the responses were received in the first few days. Additionally, the survey was intentionally conducted during a period in the fall that would minimize interference from any holidays or major conferences.

Among O'Rourke's factors, only two appear to have been candidates for affecting the response rates: subject interest or offering an incentive. Whether the subjects were interested is always up for debate. However, it should be noted that the survey dealt with an issue of professional nature, thus it might be assumed that it was of at least some interest. A similar survey was used in an earlier study and had higher response rates (Kittleson, 1997b). If offering an incentive is an issue, then health educators will need to determine how to provide some type of reward to increase response rates. If we conclude that none of these issues had an affect, then based on O'Rourke's criteria, we must look elsewhere for an explanation as to why both methods had lower response rates than past studies.

A unique issue not addressed by O'Rourke (1999) is the recent proliferation of e-mail ads. It is common for some to receive a dozen or more commercial ads a day. Perhaps the inundation of e-mails, the proliferation of spam, and the tendency to "trash" items in one's e-mail if they are not immediately pertinent, are new and crucial factors in e-mail or Web-based surveys. If this is true, health educators must think of creative methods to catch the attention of potential respondents.

Because it is still relatively early in the technology age, it may be difficult to determine what is considered an acceptable response rate. Currently, we are using comparisons to more established protocols (i.e., postal service and telephone surveys). A 40-50% response rate may indeed be outstanding when one considers the amount of information overload to which many users are exposed. For example, Sheehan (2001) reviewed dozens of e-mail-initiated surveys over the last decade. She concluded that response rates have been diminishing over time. Between 1991 and 1996, the average response rate was 46.8% across the eleven studies reviewed. Whereas, the average response rate dropped to 29.5% across the 18 studies conducted between 1997 and 2000.

IMPLICATIONS FOR PRACTICE

From the foregoing discussion, two important implications can be identified for practitioners. First, because Web and e-mail surveys appear to offer similar response rates, the choice of the data collection method should be based on the primary advantages of each (see Table 2). Researchers who are collecting non-personal, non-invasive data might be encouraged to use e-mail because it has one less step for respondents and requires less design work of the researcher. Conversely, those seeking data of a highly personal nature may be encouraged to use Web-based, data-collection where respondents may be more willing to complete such a survey where anonymity is much more transparent. Additionally, compared to telephone or postal service, both can be relatively cost effective methods of data collection depending on the amount of "set up" required and the number of respondents desired (see Figure 2 and Table 1).

Second, researchers must recognize that e-mail and Web-based data collection has experienced and will likely continue to experience diminishing response rates. Acceptable response rates with either method will require the researcher to pay careful attention to the factors outlined above. Special attention should be given to steps that would make the survey stand out.

For example:

* Offer incentives or awards.

* Format the e-mail or Web page to increase interest.

* Clearly state in the subject line the nature of the e-mail in a way that will prevent it from being trashed with the "spam."

* Enhance salience by matching respondents with survey topics that they will be highly interested in or concerned about.

Technology is increasingly becoming a more important part of the daily communication patterns of most health education professionals. The results of this study highlight the need for more research on the best ways to use the Internet for survey research. It may also be crucial to examine how health educators can expect to use the Internet to collect data from the public when many are reluctant to participant in such research themselves.

CHES AREAS

Responsibility IV--Evaluating Effectiveness of Health Education Programs

Competency A--Develop plans to assess achievement of program objectives

Sub-competency 4--Select appropriate methods for evaluating program effectiveness

Sub-competency 6--Evaluate existing data gathering instruments and processes

Sub-competency 7--Select appropriate qualitative and/or quantitative evaluation design

Competency B--Carry out evaluation plans

Sub-competency 2--Utilize data collecting methods appropriate to the objectives

Sub-competency 4--Implement appropriate qualitative and quantitative evaluation techniques

Sub-competency 5--Apply evaluation technology as appropriate

Responsibility VIII--Apply Appropriate Research Principles and Methods in Health Education

Competency B--Use appropriate qualitative and quantitative research methods

REFERENCES

Bernhardt, J. M., & Hubley, J. (2001). Health education and the Internet: The beginning of a revolution. Health Education Research, 16, 643-645.

Colorado State University. (2003). Writing@CSU: Writing Guide. Retrieved December 29, 2003 from http://writing.colostate.edu/references/research/survey/com3b1.cfm

Cook, C., Heath, F., & Thompson, R. L. (2000). A meta analysis of response rates in Web or Internet-based surveys. Educational and Psychological Measurement, 60, 821-836.

Marketing Research Association. (2002). Respondent Cooperation/Current Rates. Retrieved January 6, 2004 from http://www.mra-net.org/resources/repondent_cooperation

NTIA and the Economics and Statistics Administration. (2002). A Nation Online: How Americans Are Expanding Their Use of the Internet. Retrieved March 24, 2004 from_http://www.ntia.doc.gov/ntiahome/dn/ index.html

O'Rourke, T. W. (1999). The importance of adequate response rates and ways to improve it. Journal of American Health Studies, 15(2), 107-109.

Parker, L. (1992). Collecting data the e-mail way. Training and Development, 46(7), 52-54.

Schonlau, S., Fricker, R. D., & Elliott, M. N. (2001). Conducting Research Surveys via E-mail and the Web. Washington, DC: Rand Publications.

Sheehan, K. B. (2001). E-mail survey response rates: A review. Journal of Computer-Mediated Communication, 6(2). [Online]. Available: http://jcmc.indiana.edu/vol6/issue2/sheehan.html

Sheehan, K. B., & Hoy, M. G. (1999). Using e-mail to survey Internet users in the United States: Methodology and assessment. Journal of Computer Mediated Communication, 4(3). [Online]. Available: http:// jcmc.indiana.edu/vol4/issue3/sheehan.html

Sheehan, K. B., & McMillan, S. J. (1999). Response variation in e-mail surveys: An exploration. Journal of Advertising Research, 39(4), 45-54.

Sproull, L. S. (1986). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159-169

Web Surveyor. (2003). Traditional vs. Online Survey Methods. Retrieved December 29, 2003 from http:// www.Websurveyor.com/survey-costs.asp

Weible, R., & Wallace, J. (1998). The impact of the Internet on data collection. Marketing Research, 10(3), 1923.

Mark J. Kittleson, PhD is a Professor and Stephen L. Brown, PhD is an Assistant Professor of Health Education at Southern Illinois University. Address all correspondence to Stephen L. Brown, PhD, Assistant Professor of Health Education, Southern Illinois University, 307 Pulliam, Mailcode 4632, Carbondale, IL 62901; PHONE: [phone]; FAX: [phone]; E-MAIL: [e-mail].

Table 1--Estimated costs for various survey methods

                                                      Phone

Desired number of respondents                   400      800    1,200
Typical response rate (1), (2), (3), (4),
  (5)                                           60%      60%      60%
Invitations                                     667    1,333    2,000

FIXED COSTS
Estimated programming & testing (@ $60/hr)
  (1), (2)                                     $600     $600     $600
Estimated computer monitoring (@ $60/hr)
  (1), (2)                                       $0       $0       $0

VARIABLE COSTS
Interviews (5 min per success
  & 1 min per failure @ $30/hr) (1), (2)     $1,333   $2,667   $4,000
Printing ($0.20/invitation)                      $0       $0       $0
Non-auto bulk postage ($0.165/invitation)        $0       $0       $0
Return postage ($0.37/invitation)                $0       $0       $0
Data entry (3 min each @ $30/hr)/(others:
  10% corrections) (1), (2)                     $60     $120     $180

                                             $1,993   $3,387   $4,780

Per respondent (1), (2)                       $4.98    $4.23    $3.98

                                                       Mail

Desired number of respondents                   400      800    1,200
Typical response rate (1), (2), (3), (4),
  (5)                                           40%      40%      40%
Invitations                                   1,000    2,000    3,000

FIXED COSTS
Estimated programming & testing (@ $60/hr)
  (1), (2)                                       $0       $0       $0
Estimated computer monitoring (@ $60/hr)
  (1), (2)                                       $0       $0       $0

VARIABLE COSTS
Interviews (5 min per success
  & 1 min per failure @ $30/hr) (1), (2)         $0       $0       $0
Printing ($0.20/invitation)                    $200     $400     $600
Non-auto bulk postage ($0.165/invitation)      $165     $330     $495
Return postage ($0.37/invitation)              $370     $740   $1,110
Data entry (3 min each @ $30/hr)/(others:
  10% corrections) (1), (2)                    $600   $1,200   $1,800

                                             $1,335   $2,670   $4,005

Per respondent (1), (2)                       $3.34    $3.34    $3.34

                                                      E-mail

Desired number of respondents                   400      800    1,200
Typical response rate (1), (2), (3), (4),
  (5)                                           30%      30%      30%
Invitations                                   1,333    2,667    4,000

FIXED COSTS
Estimated programming & testing (@ $60/hr)
  (1), (2)                                   $1,200   $1,200   $1,200
Estimated computer monitoring (@ $60/hr)
  (1), (2)                                     $240     $240     $240

VARIABLE COSTS
Interviews (5 min per success
  & 1 min per failure @ $30/hr) (1), (2)         $0       $0       $0
Printing ($0.20/invitation)                      $0       $0       $0
Non-auto bulk postage ($0.165/invitation)        $0       $0       $0
Return postage ($0.37/invitation)                $0       $0       $0
Data entry (3 min each @ $30/hr)/(others:
  10% corrections) (1), (2)                    $600   $1,200   $1,800

                                             $2,040   $2,640   $3,240

Per respondent (1), (2)                       $5.10    $3.30    $2.70

                                                       Web

Desired number of respondents                   400      800    1,200
Typical response rate (1), (2), (3), (4),
  (5)                                           30%      30%      30%
Invitations                                   1,333    2,667    4,000

FIXED COSTS
Estimated programming & testing (@ $60/hr)
  (1), (2)                                   $2,400   $2,400   $2,400
Estimated computer monitoring (@ $60/hr)
  (1), (2)                                     $240     $240     $240

VARIABLE COSTS
Interviews (5 min per success
  & 1 min per failure @ $30/hr) (1), (2)         $0       $0       $0
Printing ($0.20/invitation)                      $0       $0       $0
Non-auto bulk postage ($0.165/invitation)        $0       $0       $0
Return postage ($0.37/invitation)                $0       $0       $0
Data entry (3 min each @ $30/hr)/(others:
  10% corrections) (1), (2)                     $60     $120     $180

                                             $2,700   $2,760   $2,820

Per respondent (1), (2)                       $6.75    $3.45    $2.35

1 = Schonlau, Fricker, and Elliot

2 = Web Surveyor

3 = Herbert Research

4 = Marketing Research Association

5 = Sheehan

Table 2. Relative Strengths/Weaknesses of Survey Methods

                                          Phone             Mail

response rates (1), (2)               highest (~60%)   moderate (~40%)
cost--small surveys (1), (2)           next lowest         lowest
    --large surveys (1), (2)             highest        next highest
instrument design/management (1)         depends            easy
data manipulation/analysis (1)           depends          most work
speed of receipt (1), (2)                fastest            slow
clarification & probing (1)               strong          very weak
long questions                             weak            strong
respondent convenience (1)                 low            moderate
interviewer bias (1)                       most             less
respondent competence needed (1)          lowest             low
missing data (2)                           best             poor
difficult patterns                         good             poor
technical difficulties (1), (2)            low               low
confidentiality/sensitive questions       lowest            good
gereralizability (1)                  good potential   good potential

                                           Email              Web

response rates (1), (2)               marginal (~30%)   marginal (~30%)
cost--small surveys (1), (2)           next highest         highest
    --large surveys (1), (2)            next lowest         lowest
instrument design/management (1)           easy          most involved
data manipulation/analysis (1)           some work        least work
speed of receipt (1), (2)                  fast              fast
clarification & probing (1)              very week           weak
long questions                            strong            strong
respondent convenience (1)                 high              high
interviewer bias (1)                       less              less
respondent competence needed (1)         moderate           highest
missing data (2)                           poor              fair
difficult patterns                        poorest            best
technical difficulties (1), (2)          moderate           highest
confidentiality/sensitive questions       better             best
gereralizability (1)                   some question     some question

(1) = Writing at CSU

(2) = Schonlau, Fricker, and Elliott

Figure 1. Response rate of email and web surveys by day

          Responses

         Email   Web

Day 1     55     70
Day 2     10     14
Day 3      4      7
Day 4      0      7
Day 5     14      2
Day 6     15     13
Day 7      1      3
Day 8     22      3
Day 9      5     19
Day 10     3      5

Note: Table made from bar graph.