Getting wired: exploiting the Internet for the collection of valid sexuality data.

The creation of the Internet has arguably been the greatest advance in communication since the advent of the telephone in 1876. It is estimated that between one third (National Telecommunications and Information Administration and U.S. Department of Commerce [NTIA & USDC], 1999) and two thirds (ACNielsen, 2000) of Americans utilize this technology, and this number is growing at an astounding rate (NTIA & USDC, 1999). Every month the population of Internet consumers is coming closer to resembling the demographics of America as a whole (United States Internet Council [USIC], 1999). The use of this technology for the collection of sexuality data represents an exciting new frontier for researchers. The anonymity and accessibility of the Internet allow data collection from samples that may otherwise be unreachable, especially for research in the sexuality arena. The ease of recruitment and data collection allows extensive samples to be readily achieved. Furthermore, those samples may be more representative, on some variables (e.g. geographically, demographically, sexually, etc.), than those obtained through traditional methodologies. Although benefits exist, several important risks must be confronted to assure the collection of valid data. Many of these potential problems are issues that must be faced with any data collection from human subjects, such as lying and sabotage. Other issues are more particular to Internet-based research, such as repeated submission of data from the same individual and greater loss of control over stimulus presentation.

The purpose of this article is to discuss these and other issues in the context of a large Internet-based study of sexuality and sexual orientation, with the goal of explaining how to maximize utility as well as combat potential risks and ethical dilemmas. Special attention will also be given to the use of the Internet to collect data from traditionally underrepresented groups (gay, transgender, paraphilics, etc.). The article ends with a look to the future of sex research over the Internet, exploring the collection of nonsurvey data, including cognitive tasks and psychophysiological assessment. Because a body of literature has already addressed using the Internet for collecting general social science data (Baron & Siepmann, 2000; Birnbaum, 2000; Hewson, Laurent, & Vogel 1996; Michalak & Szabo, 1998), this article focuses more specifically on the collection of sexuality data.

To help clarify many of the issues and recommendations presented in this article, each will be explained in the context of a large-scale Internet-based study of how gay, lesbian, and bisexual (GLB) college students define "having sex" (Mustanski, 2001). The purpose of this Internet-based research project was to explore the existence of both between- and within-group semantic heterogeneity in how GLB individuals define having sex, for the purpose of improving sexuality survey research, clinical communication, and public health efforts. Participants were recruited by e-mailing the majority of university GLB student organizations in the U.S. and asking them to promote the websurvey, producing a national sample of 1,536 homosexual and bisexual men and women. University students were targeted to make the sample comparable to past research with heterosexual college students (Sanders & Reinisch, 1999). The webpage was created by the author using Microsoft FrontPage and hosted by the University server, and all data was collected anonymously. The survey contained a number of different sexual behaviors to which participants responded if they would classify the activity as having sex. The results of the study suggested that homosexual individuals vary widely in how they classify having sex. Significant differences based on sex and sexual orientation were also identified. More information about this study, such as specific problems that were encountered and how they were overcome, will be provided as examples in the following sections when relevant.

The following list of pertinent issues has been compiled based on the author's own experience conducting an online sexuality study, as well as by adapting suggestions from nonsexuality Internet-based research. Each topic contains advantages, disadvantages, and suggestions for overcoming potential problems.

ISSUES RELATED TO INTERNET BASED SEXUALITY RESEARCH

Cost

Many Internet-based research projects can be conducted with few, if any, related expenses. Because most universities provide free web hosting for affiliated individuals, the server used for data collection may be cost free. If increased control or security is desired, most personal computers can be used as web servers with the addition of inexpensive software. With the advent of easy-to-use website design software, researchers can create web sites for most survey studies themselves, without having to hire a computer programmer. Outflows related to printing, mailing, or coding can be eliminated by recruiting participants via e-mails or web-postings and by having data automatically entered into a database. The downside of utilizing University resources is that there may not be people available who are knowledgeable enough about the Internet to consider the ethical and methodological issues associated with this research.

The fact that the majority of Internet-based studies have been conducted without offering participant incentives (Musch & Reips, 2000), instead relying on participant altruism, suggests that subject payment may not be necessary when using this methodology. However, differences between volunteers and participants who received incentives have been demonstrated in non-Internet based research (Tomporowski & Simpson, 1993 but also see Klein & Higgens, 1992). Participation rates also appear to increase when incentives are offered (Blomberg & Sandell, 1996; Maughan & Higbee, 1981; Wiseman, Schafer, & Schafer, 1983), suggesting that lack of incentives may produce a self-selection bias in data collection. More recently, an online experiment randomly assigning participants to an incentive or incentive-free condition suggests that incentives do influence drop out, but not initial interest in participation (Frick, Bachtiger, & Reips, 2001). Interestingly, the drop out in both conditions was relatively small (18.5% versus 9.5%), and providing incentives did not alter how participants answered the questions.

It is essential for researchers considering conducting on-line research to consider possible methods of reducing participation bias. One option is to offer either a direct monetary incentive or a lottery-style incentive. The problem with offering a direct monetary reward is that it requires participants to reveal their identities so that the reward can be delivered. Donations to a charity for each submission is another possibility that does not require participants to reveal their identities, although this type of incentive schedule may result in multiple submissions by the same individual in an attempt to garnish a larger donation. The same problem exists when a lottery system is used. Either the participant needs to reveal their identity, which compromises anonymity, or they may submit multiple times in order to increase their chances of winning. Additionally, certain federal, state, and university laws may come into play by even the mention of the word "lottery." To avoid these issues it is possible to provide other forms of incentive such as feedback based on the participant's results, which may be considered rewarding for subjects who chose to participate out of curiosity. This option may not be relevant or ethical for all designs, however, because some may have no feedback to offer while others cannot ethically provide the information in such a context.

Another option, and the one used by Mustanski (2001) in the on-line study of the definition of having sex, is to conduct Internet-based research with a population interested in being studied, thus precluding the necessity of extrinsic incentives. In this study, many gay and lesbian individuals commented that they enjoyed participating and looked forward to inviting their peers to participate because they found the study to be stimulating and interesting. Many expressed satisfaction about the fact that their population was being given serious study. Fortunately, sex is a subject that many Americans are interested in discussing, especially on the Internet (Mills, 1998), thus making it a topic likely to induce participation from many individuals. If no incentives are being offered, the length of the study becomes critical since participation declines as a study becomes more aversive (Korn & Hogan, 1992), possibly explaining why the majority of internet-based studies required less than 15 minutes for participation (Musch & Reips, 2000). Further research is needed to examine the effects of offering incentives in on-line research.

Ease of Use and Lack of Experimental Control

Another attractive feature of conducting research via the Internet is the ease with which the data are collected. Because participants conduct the study on their own computer whenever they desire, there is no need for scheduling subjects, hiring interviewers and coders, or spending time conducting the research. Once the study is set up, subjects can complete the study any time of day or night, if allowed, without any further effort by the researchers.

As with any research that takes place outside of a laboratory, the drawback is that it is difficult for the researcher to control the environment in which the individual is completing the study. If the design of the study requires standardized stimulus presentation, such as visual or auditory stimuli, using the Internet to collect data becomes much more complex. Variability in monitor specification, Internet connection speeds, and pop-up banner advertisements are just some of the possible factors that could confound standard stimulus presentation. Even for research that does not require participants to perceive highly specific stimuli, the quality of the data are potentially decreased because Internet-based research does not take place in a controlled environment. Some subjects may participate in a distracting environment--late at night when they are not fully alert, or while intoxicated--thus threatening the validity of the data. For some sexuality research it is important to control for the sexual arousal level of participants, something that will likely vary considerably depending on which previous websites the participants have visited. Differences in level of sexual arousal can be expected between participants who arrive at a study after reading a general news website versus one that presents erotic materials. Although it is possible for the researcher to request that subjects fulfill certain requirements, which may help to standardize the environment, there is no guarantee that they will be adopted. Researchers may also inquire about the environment in which the study took place, but it may be difficult to determine how to utilize this information. It is important to keep in mind that many of these problems have an equal probability of occurring even when other survey methodologies are employed.

Sample Size

With hundreds of millions of people connected to the Internet around the world (USIC, 1999) the potential sample size far exceeds that of the Introduction to Psychology subject pools often used in sexuality research. Although no one would expect millions of participants in an on-line study, numbers in the thousands are not uncommon (Musch & Reips, 2000). Thousands of individuals visit the Web's Experimental Psychology Lab each month (Reips, 2000), and the previously cited study of the definition of having sex received over 3,200 visitors in 2 months, resulting in 2,800 completed surveys (Mustanski, 2001). Sexuality researchers should be particularly interested in becoming active in online data collection efforts because of the large interest in sexual materials and communication via the Internet, so much so that the Internet has been labeled an erotic oasis (Ross, Tikkanen, & Mansson, 2000). Sex related searches predominate the list of most common search terms on the web (see http://www.searchterms.com/, accessed 18 July 2000, and http://www.searchwords.com/top100w.htm, accessed 18 July 2000) and the alt.sex newsgroups and sex related chat rooms are extremely popular (Binik, Mah, & Kiesler, 1999). This increase in the number of participants has the potential to boost the power of sexuality research studies (Cohen, 1992).

Because web servers automatically record the number of visits to a site, it is possible to subtract off the number of respondents in order to calculate the number of nonparticipants. This figure can be estimated more accurately if the number of recruited subjects is known, allowing the calculation of both the number of total nonresponders as well as the number of individuals who visited the site but did not participate. The ability to calculate these statistics allows researchers greater knowledge about their pool of nonresponders than is allowed in traditional survey research.

External Validity 

Researchers expect that the demographics of Internet users will rapidly approach those of the general population (Kehoe & Pitkow, 1996), with 70% of teenagers already accessing the Net regularly (Grunwald Associates, 2000). Males and females are already present on the Internet in representative proportions (Nielson//NetRatings, 2000). Gay and lesbian people are also very well represented both inside and outside of the U.S., and their rates are growing rapidly (Computer Economics, 2000). Some ethnic minorities (Black and Hispanic) and people of low socioeconomic status or education, on the other hand, are not as well represented as Caucasians, wealthier individuals, and the better educated (NTIA & USDC, 1999), although the racial, educational, and socioeconomic gaps may be narrowing (NTIA & USDC, 2000; USIC, 1999).

Even if Internet users are not currently representative of the U.S. population, they may be more representative on some variables then the typical college student who is the subject of most research. In his review of the web experiment methodology, Reips (2000) points out that McNemar (1942, p. 333) has dubbed the human behavioral sciences "the science of the behavior of sophomores," with over 80% of psychological studies being conducted on students (Reips & Bachtiger, 2000; Shultz, 1972). The demographic variability of self-selected participants in Internet-based research has been found to be greater than in most laboratory studies (Reips, 2000), and one of the only published on-line sexuality studies reported a more diverse set of individuals in terms of age and more proportional number of males and females (Bailey, Foote, & Throckmorton, 2000). Although this study found some demographic differences between the college and Internet samples, they were not found to differ on the majority of behavior characteristics, and only moderately on some attitudinal variables (Bailey et al., 2000). In the Internet-based study of the definition of having sex, participants had a larger age range and more geographic diversity than are typically found in studies conducted in laboratories on campus (Mustanski, 2001). Krantz and Dalal (2000) reviewed findings from Internet-based research in a variety of areas and found a close correspondence between data collected via the Internet and that collected in the laboratory. Taken together, these data suggest that although more diverse samples can be reached via the Internet, the majority of research results appear to remain consistent.

The degree to which a study is externally valid is largely determined by the recruitment methodology utilized. Just as in traditional research, two main approaches exist: probability and convenience sampling. Couper (2000) provides a thorough review of the survey error related to various types of sampling methods currently available on the Internet. No sexuality studies have yet to be published utilizing true probability sampling, which is most likely a result of the prohibitive expense involved. In past Internet-based research, a variety of different forms of media have been used to recruit convenience samples, including promoting the experiment on the web or in newsgroups, sending targeted e-mails, and registering the research webpage with search engines, as well as advertising in print media and over the radio (Musch & Reips, 2000). Getting a link for the research webpage on a major Internet portal or a site visited by a large number of people relevant to the research question may work well depending on the target sample. Chat rooms and e-mail distribution lists are another option that should be considered, but caution should be used when generalizing from any one such media to all members of a population. Recruiting across several such mediums may decrease the possibility of biases occurring.

Perhaps the most common error made by researchers attempting to conduct valid research on the Internet is creating a study on-line, and then registering the URL with a number of search engines and collecting data from anyone who arrives at the study website. This is the equivalent of placing a study in the yellow pages under the heading "sexuality surveys." When this method is used, the researcher should consider what type of person is searching for keywords like sex survey, sexuality research, or in the case of my previous study, having sex. Are these individuals who should be recruited into the study, or are there major self-selection biases? Are these self-selection biases any worse than those found in traditional survey research?

To avoid this issue it is advisable to proactively recruit using methods that target individuals based on the research questions. Posting to relevant newsgroups, listserves, or webpages are several such methods. Information is so accessible on the Internet, however, that even when proactive methods are used it is likely that data will be received from populations that were not directly recruited. In this case it is wise not to use these data because they are probably not representative of their population. In other words, if recruitment efforts are targeted at heterosexual Caucasian women and data is received from a number of homosexual Asian men, those data should be included in analysis with caution, and by no means should be taken as representative of that population.

The submission of data from individuals within the target population who were not directly recruited has both benefits and risks. As discussed in the next section, beyond simply increasing the sample size, referral recruitment may allow a more representative sample of the target population because it allows participation from otherwise unreachable subjects. On the other hand, referral recruitment, also called snowballing (Goodman, 1961), may introduce new biases into the data when it is the sole sampling method used. It has been suggested that snowballing may undersample social isolates, those of low education, social class, or income, as well as social deviants (Welch, 1975). Additionally, biases may be introduced because the characteristics of the initial sample may be over represented, because the referrers are uncharacteristically altruistic, or because of potential "masking" effects from peers not being referred because of privacy concerns (Heckarthon, 1997). Empirical research has not supported many of these concerns (Etter & Perneger, 2000; Welch, 1975), and the use of the Internet may overcome others. Because participants can refer peers anonymously, masking effects can be avoided. Since socially isolated and deviant individuals are well represented on the Internet, these methods may sample them appropriately if not excessively.

The benefit of referral recruitment can be maximized by making it simple for participants to provide information about the study to their peers. One way to do this is to have all the necessary contact information on the final webpage of the study, and remind the participants to recruit others who are from the target population. Another option would be to provide a form where the participants could enter the names of peers they would like to recruit, and to have a computer program automatically e-mail these individuals and ask them to participate. Using the latter method presents several anonymity and ethics issues that should be given consideration before this method is applied.

Using different URLs for each recruitment effort is advisable to help determine if there are any group differences based on how participants are recruited into the study. Reips (2000) refers to this as the multiple site entry technique. To do this, set up a number of different entry pages with different URLs, all of which can have the same content, and make sure to track from which URL the data was submitted. Then, when recruiting from different media, make sure to give each a different URL. During data analysis the results from the two groups should be compared to look for systematic differences. For example, if recruiting from both chat rooms and USENET groups, give one URL to the chat room sample and post a different one to the USENET group. It is also possible to include an item assessing how the participant found out about the study. Such an item was included in the having sex study (Mustanski, 2001), but resulted in responses that lacked sufficient clarity to be useful.

Until the Internet data collection methodology is refined and matures it is also advisable to compare the Internet data to traditional information sources, such as random samples and laboratory convenience samples, in order to examine sample biases. The collection of demographic data from both groups will allow an analysis of possible response correlations with demographic information and demographic and response differences between samples, and determine whether any general trends remain after adjusting for demographic differences. These data will also allow researchers to estimate how representative their sample is of the population of interest. Internet sexuality research is still in its infancy, and even though current research suggests exciting possibilities, more research on this methodology must be conducted before it is possible to definitely conclude that Internet-based samples are externally valid. In the meantime, national random samples are still the only source of base-rate statistics.

As mentioned previously, in the study of the definition of having sex for GLB people, participants were recruited by e-mailing every GLB student organization in the U.S. (Mustanski, 2001). The site was not registered with any search engines, and code could have been written into the page to prevent the site from being listed. This method was used to maintain the integrity of the sample. Because homosexuals are a stigmatized group in society, there was concern in conducting the on-line study that sexually prejudiced individuals might attempt to sabotage the research. By recruiting directly from GLB organizations, asking these participants to recruit other GLB individuals, and not registering it with search engines, it was hoped that the project would only become known to GLB peoples. This method appears to have been successful because no malevolent submissions or e-mails were received. Instead, a large number of individuals indicated they were recruited into the study by a friend or their university GLBT organization and were planning on passing on word of the study to other peers. Additional anecdotal evidence for referral recruitment came from the fact that even though the first phase of the recruitment effort only included universities on the west coast, within hours submissions from all over the country were received. Empirical research exploring the effects of various recruitment methods utilizing the Internet should be undertaken in the future.

Utilizing Virtual Communities to Recruit Underrepresented Populations

In the same way that the Internet can produce a more diverse and potentially representative sample, it can also allow for the collection of data from subjects who might not otherwise be reached. Survey research often suffers from large numbers of nonresponders or dropouts who are uninterested in being identified, and whose absence can lead to invalid conclusions (Birnbaum & Mellers, 1989). This issue becomes increasingly important for sexually stigmatized groups such as homosexuals, individuals in the Sadomasochist/Bondage/Discipline (SMBD) community, or individuals with paraphilias. The anonymity of the Internet may help to solve this problem by allowing subjects to participate without being identified.

Beyond increasing response rates, the Internet may make sexual minorities easier to recruit because many of these individuals have formed "virtual communities" (Rheingold, 1993) for discussion, support, networking, and entertainment purposes. Many sexuality-based groups such as homosexuals, transsexuals, people with fetishes, or those with nontraditional sexual habits maintain web pages and listerves for their members to exchange information. For example, Gay.com, a website catering to GLBT individuals, has approximately 2.5 million visitors a month (M. Elderking, personal communication, July 20, 2000). Another example, www.annelawrence.com, is a clearinghouse for information relevant to transsexual individuals that receives approximately 90,000 visitors per month. In 1997, when the site was much less popular, an on-line survey of transsexuals resulted in 240 completed submissions in 40 days (A. Lawrence, personal communication, January 17, 2001). The alt.sex.fetish newsgroups are extremely popular and cater to just about any particular sexual interest. Although no single website or newsgroup is likely to be representative of a target population, recruiting across several different platforms may diminish this issue.

Anecdotal evidence for the willingness of such organizations to help spread the word about on-line research relevant to their organizations came from the large numbers of participants in the having sex study (Mustanski, 2001) who reported having heard about the project through an organization's listserv or even through chat rooms. Many of these individuals might not have been reached with traditional survey methods because they may not have' "come out" yet, thus making them unrecruitable. One study of GLBT youth via the Internet suggests that the majority of youth who have access to the Internet come out on-line before doing so in real life (!OutProud!/Oasis, 1998). In approaching these organizations for help in recruiting participants, the author found it important to be brief but informative, explain the nature and importance of the study, and make it clear that these organizations are very important in the research process. Offering them access to the results and approaching them with the attitude that they are collaborators in a very important research project also seemed to be advantageous.

Internal Validity 

Just as in research on other sensitive topics, sexuality researchers must be vigilant about the accuracy and honesty with which their subjects respond to study questions. In his review of conceptual models dealing with self-presentation bias in interview situations that focus on assessment of human sexuality, Catania (1999) presents a body of research suggesting that a participant's degree of discomfort is positively correlated with nonresponsiveness (Catania, McDermott, & Pollack, 1986) and negatively correlated with honesty (Catania et al., 1996). Furthermore, assurances of anonymity reduce discomfort and attempts to manage impression (Agnew & Loving, 1998). Although there is increasing concern about privacy on the Internet (Roper Center, 1999a), a majority of American Internet users are still willing to provide personal information over the Internet, especially when their anonymity is assured (Roper Center, 1999b).

Internet-based interactions also appear to be characterized by disinhibited behavior (Joinson, 1998) causing people to be more willing to swap personal information (Parks & Floyd, 1996) or publish normally protected information about themselves (Joinson, 1998). Research has found that participants using the Internet respond with less social desirability and social anxiety (Joinson, 1999), thus increasing the potential for honest self-report. On the other hand, Joinson (1999) suggests that lying may be increased by the anonymity, increased disinhibition (Joinson, 1998), and increased deindividuation (Kiesler, Siegal, & McGuire, 1984) found on the Internet. According to deindividuation theory, in such a context there will be increases in hostility and reduced self-regulation (Zimbardo, 1969). Therefore, the same anonymity thought to reduce social desirability may backfire and actually decrease honesty (Joinson, 1999). An example of this may be the creation of "manufactured identities" (Rheingold, 1993), which are characters created by users to present themselves to others on the Internet. Although some empirical work seems to suggest that in these contexts social desirability really does decrease (Joinson, 1999), researchers interested in conducting sexuality research via the Internet must be aware of the potential for decreased self-regulation of their participants and take steps to avoid it. One possibility is to include a social desirability questionnaire such as the Marlowe-Crowne Social Desirability Scale (Crowne & Marlowe, 1960) or the Balanced Inventory of Desirable Responding (Paulhus, 1988). Just as with any survey research the inclusion of methodologies for detecting inappropriate responding should also be included to help identify participants whose data should be excluded. One example is to include similar questions with the affirmative response switched to identify participants who are responding to the questions randomly. Another option is to include items that everyone would endorse, and check to see that the participant did, in fact, respond appropriately.

Beyond inappropriate responding, potential sabotage of Internet-based research is a threat to internal validity. When studies are conducted anonymously on-line it is possible for the same individual to participate multiple times, thus potentially skewing the results. The probability of this occurring varies with the population under study, the method used to recruit and encourage participation, and prevention methods taken. Several solutions to this problem have been suggested, such as asking participants not to participate more than once, asking them about the seriousness of their answers or if they have already participated, or collecting their e-mail or IP-address (Reips, 2000). Unfortunately, all of these methods are problematic. Asking participants not to participate more than once, or if they have already participated, does nothing to screen out deliberate sabotage. It may, however, help prevent the same individual from accidentally or ignorantly participating multiple times. Asking participants for their e-mail address is problematic because it removes the anonymous nature of the study. Because most Internet service providers now use dynamic assignment of IP addresses, which assigns the user a new IP address with each connection, it is not possible to track participants across connections using this methodology. Additionally, many public computer facilities use Ethernet connections to the Internet, in which each computer is assigned a static IP address. In this case, if repeated submissions are screened out based on IP address, different participants who use the same computer in a public facility will be disqualified. "Cookies," which are files placed on the participant's local computer, can be used to track individuals across connections but there may be ethical issues with their use (Reips, 2000). Cookies also provide only a partial solution, since they will confuse different participants using the same computer with multiple submissions from the same individual. The good news is that Internet-based research that took place before dynamic addressing suggests that repeated submissions by the same person are an extremely rare occurrence (Reips, 2000).

Violating the rules of on-line etiquette, also called netiquette, can also increase the probability of sabotage occurring. Consequences for not following netiquette include spamming, or flooding a user with repeated e-mails; false responses; repeated submission by the same individual; and hacking. McMurdo (1995) provides a digest of general netiquette advice while Shea (1994) provides rules that should be followed in any on-line communication, such as respecting people's time, bandwidth (the amount of information [bits] that can be passed along an Internet connection in a given period of time), and privacy, as well as closely following discussion group rules. Binik et al. (1999) bring up the importance of solicitation advertisements telling participants about compensation, timing, recruitment criteria, identifying researchers and their affiliations, and being in "good taste." Using erotic visual materials for recruitment purposes should be done with caution as it can put researchers or their institution at risk for prosecution (Binik et al, 1999). Using unsolicited e-mails can also be problematic, as pointed out by Mehta and Sivadas (1995), who received many complaints for this and, therefore, concluded it is an unacceptable practice. Binik et al. (1999) point out that unsolicited e-mails for sex research may be particularly problematic because many people read e-mail at work, where the content may be monitored by their employer, and also because many people share e-mail accounts with family members including life-partners and children. In the study on the definition of having sex (Mustanski, 2001) no complaints were received about unsolicited e-mails, which suggests that the recruitment message, the method of e-mail solicitation, and the intended recipients may moderate the acceptability of such recruitment methodology.

Because it is difficult to ascertain a subject's motivation for participating in an on-line study, researchers should be cautious about informing potential participants that they are ineligible for a study because of particular characteristics. If the prospective subject is particularly curious about the contents of the study, or has little interest in following the rules set out as criteria for participation, they may falsify characteristics that would exclude them. To get around this problem it is preferable to simply ask participants about all relevant characteristics and then filter out those who do not meet the study requirements.

The internal validity of research conducted via the Internet may also be increased by the removal of interviewer effects (sex, race, sexual orientation, etc.), which are a particularly sensitive issue in sexuality research (Catania, Binson, Van Der Straten, & Stone, 1995). Although prior studies have found mixed results for the influence of interviewer effects in sexuality research (Catania et al, 1995), the ability to control for this factor in Internet-based studies is a clear benefit. Whether or not interviewer effects are ameliorated to a greater extent over the Internet than in traditional mail survey research is an empirical question that should be addressed.

Comparing Internet and Traditional Research Samples

Although the idea of controlling for any potential confounds in a quasi-experimental between-subjects design may seem fundamental, the methodological aspects of translating this rule appropriately for comparisons between Internet and traditional samples has yet to be resolved. For example, a recent article by Ross et al. (2000) compares the differences between Internet and conventional samples of men who have sex with men. The Internet sample participated anonymously through the webpage of the Swedish federation for lesbian and gay rights (RFSL). The conventional, written survey was distributed using very different means. Copies of it were made available at three branch offices of RFSL found in large cities, as well as being distributed through key members of the gay community throughout Sweden. Based on these samples the authors conclude that gay Interact responders are younger, are more likely to live in smaller cities, have lower formal education, and have less sexual experience solely with other men; thus, there are a set of biases for Internet data which distinguishes it from conventional samples (Ross et al., 2000). The problem with this conclusion is that there were other differences between the two samples besides the method by which the data was collected. The Internet-based sample was assured anonymity throughout all stages of the data collection process, had potentially different motivations for going to either the RFSL webpage or into an office, and were not required to know any key members of the gay community. Additionally, the fact that the RFSL offices used to distribute the traditional survey were all located in larger than average Swedish cities is just as likely to explain the differences between the two groups as the differences in research methodology.

A better way to conduct a study comparing Internet and conventional samples would be to ensure that the samples differ solely on the research medium being utilized. An example of this would be to conduct a study similar to the one on how GLB college students define having sex. All organizations would be randomly assigned to receive either the Internet version or a mailed-out version of the questionnaire. The mailed-out version would include postage-paid surveys sent to group leaders to distribute to their members, and for those members to distribute to their peers. The two groups would be of comparable age, education, affiliation with GLB organizations, level of anonymity, and several other characteristics that could also potentially explain any differences. The factor structure and demographic information could then be compared to look for differences between the two samples.

Ethical Issues

Binik et al. (1999) provide an overview of ethical issues in conducting sexuality research via the Internet. Researchers must remember that depending on the methodology used, materials placed on the Internet are potentially accessible to anyone, including minors. For some fields this is a non-issue because the measures used are innocuous, but in the sexuality arena there is the potential that some guardians would object to minors observing sexuality research materials. The prevailing attitude seems to be that it is the responsibility of guardians to shield minors from accessing materials they deem offensive as long as the website's owners have made a reasonable attempt to prevent them from reaching the site. Researchers and their Human Subject Review Boards must evaluate what constitutes such reasonable attempt.

It is important to recognize that obtaining informed consent and receiving human subjects committee approval is an important part of any research project, and collecting Internet-based data is no exception. How to go about this within the confines of current technology is still a matter under debate. Binik et al. (1999) give the example of having participants type their name into a form to indicate they have given consent, but this is problematic because it removes anonymity, even if it is just perceived loss of anonymity. In the study of GLB people's definition of having sex (Mustanski, 2001), following the informed consent information was the statement, "I have read this form and have had all my questions answered to my satisfaction. The submission of a complete survey constitutes my consent. Only persons over 18 should participate." Following this was a link, which took the participants to the study questions. The full informed consent statement can be accessed at http://www.indiana.edu/~sexquest. Binik et al. suggest alternate methods of verifying informed consent via the Internet, such as authenticating the participant's information with a telephone call or crosschecking with some other information source.

Web Page Design

The aesthetic of the web page is important for convincing participants that the research is legitimate and important. A well-laid-out site that is easy to follow and formal in appearance will inspire confidence in potential participants. Hosting the survey on a university or departmental server will add legitimacy over hosting it on a personal computer or commercial server. Adding a departmental logo to the site will likely provide additional confidence.

When writing the code for a webpage, researchers should consider adding several additions beyond that which produces the visual page. For example, code can be written to prevent an individual from copying the entire research page and reposting it at another location, with the possibility of altering the format or questions. Since many free CGI/Pearl scripts for form processing often do not check for posting from other domains, if precautions are not taken this type of sabotage could occur without the researcher's knowledge.

Another consideration is preventing the listing of research webpages by search engines when the research question necessitates they only be available to a special population. For example, consider a study where the Internet is being used to collect outcome data on the effects of a particular drug on women's sexual functioning, and the subjects are invited to participate by their prescribing physicians. In this case it would be wise to prevent nonprescription users from participating, but because of the recruitment method the use of passwords may not be possible. Less Internet-savvy researchers may assume that a webpage will not be listed if it is not registered with search engines, and thus will be unavailable to the general public. This, however, is not the case, as search engines now use "robots" to search the web and catalogue its contents. To prevent this from occurring to a research webpage, code can be written into a robot.txt file (see http://www.kollar.com/robots.html for instructions) or through the use of Meta tags (see http://www.lycos. com/help/robots.html#meta for instructions).

The number of pages used to collect data is another important consideration. One long page may seem confusing or overwhelming, while multiple pages may increase the download time and not allow participants to evaluate their commitment at the start, thus potentially increasing dropout rates. On the other hand, using multiple pages allows the collection of data from participants who do not complete the entire study. Partial data cannot be collected from single-page forms except in the unlikely case the user clicks the submit button after electing not to complete the study. Reviewing other studies currently being conducted on the Internet will provide some assistance in addressing this and other similar design issues (see appendix for list of relevant websites).

Lastly, the website should be pretested on different computer platforms (Macs and PC) and with different software (Internet Explorer and Netscape), as well as on different screen resolutions and monitor sizes to make sure that the page is robust across all of these variables. Pretesting the program that manages the data is also essential because several problems have been reported, such as commas entered in a comment section by participants ruining the column structure of the data file.

CONCLUSION

The Internet is an exciting new frontier for the collection of sexuality data. Currently, the majority of research conducted on the Internet has consisted of surveys based on convenience samples (Musch & Reips, 2000). A few genuine experimental studies have been conducted in the areas of perception (Hecht, Oesker, Kaiser, Civelek, & Stecker, 1999), decision making (Birnbaum, 1999), personality (Buchana & Smith, 1999), psychoacoustics (Welch & Krantz, 1996), and cognitive tests requiring the collection of reaction time data (McGraw, Tew, & Williams, 2000). Future researchers should consider the possibility of exploiting the Internet data-collection method to its fullest. Several commercial companies now allow true probability sampling to be achieved on the Internet. Programs can be created to present stimuli, allow participants to perform a number of tasks, randomize entering participants into experimental groups, provide real-time data checks, and include automated branching based on responses. Using this technology, participants can be grouped based on responses to questionnaire items, and then presented with differing stimuli based on particular research questions. Cognitive and perceptual assessments can also be conducted using these tools, allowing a number of experiments such as those examining differences in the cognitive performance of men and women. With scanners becoming more and more common it will be possible to conduct research on physical characteristics like finger length in homosexuals compared to heterosexuals (Williams et al., 2000). The use of microphones may allow the collection of data such as heart rate or vocal data, and digital video cameras for the collection of visual data. Interactive multimedia technology allows remotely located individuals to interact across distances, as well as allowing the researcher control over the interaction. Recent advances in the area of telemedicine suggest that it is possible to collect reliable psychophysiological data remotely (see http://www.telemedtoday.com/), although much more research must be done to adapt these tools for research purposes and allow them to be used cost-effectively. As technology becomes more advanced, and bandwidth increases, many of these issues may be resolved. That, coupled with the increasing numbers of people connecting to the Internet and their on-line interest in sexuality makes this medium an exciting one for the collection of sexuality data in the future. It is important to remember, however, that although the use of the Internet to collect sexuality data has many potential benefits, risks to experimental validity must also be combated to insure the collection of quality data. Many of these risks also need to be put in the perspective of current limitations inherent in traditional research methodology. Further methodological research is clearly needed to assure the collection of valid data as sex researchers begin to explore this new frontier.

APPENDIX

Examples of Psychological Experiments on the Internet

* PsychExps: http://www.olemiss.edu/PsychExps/

* Experimental Psychology Lab: http://www.psych.unizh.ch/genpsy /Ulf/Lab/WebExpPsyLab.html

* Psychological Research on the Net: http://psych.hanover.edu/APS /exponnet.html

* Yahoo's listing of Psych Experiments: http://dir.yahoo.com/Social_Science/Psychology/Research/ Tests_and_Experiments/

REFERENCES

ACNielsen. (2000). ACNielsen survey finds nearly two thirds of U.S. population age 12 or older are online. Chicago, IL. Retrieved January 5, 2001 from the World Wide Web: http://acnielsen.com/news/american/us/ 2000/20000508.htm.

Agnew, C. R., & Loving, T. J. (1998). The role of social desirability in self-reported condom use attitude and intentions. AIDS and Behavior, 2, 229-239.

Bailey, R. D., Foote, W. E., & Throckmorton, B. (2000). Human sexual behavior: A comparison of college and Internet surveys. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 141-168). San Diego, CA: Academic Press.

Baron, J., & Siepmann, M. (2000). Techniques for creating and using web questionnaires in research and teaching. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 235-265). San Diego, CA: Academic Press.

Binik, Y. M., Mah, K., & Kiesler, S. (1999). Ethical issues in conducting sex research on the Internet. The Journal of Sex Research, 36, 82-90.

Birnbaum, M. H. (1999). Testing critical properties of decision making on the Internet. Psychological Science, 10, 399-407.

Birnbaum, M. H. (2000) Psychological experiments on the Internet. San Diego, CA: Academic Press.

Birnbaum, M. H., & Mellers, B. A. (1989). Mediated models for the analysis of confounded variables and self-selection samples. Journal of Educational Statistics, 14, 146-158.

Blomberg, J., & Sandell, R. (1996). Does a material incentive affect response on a psychotherapy follow-up questionnaire? Psychotherapy Research, 6, 155-163.

Buchana, T., & Smith, J. (1999). Research on the Internet: Validation of a World-Wide Web mediated personality scale. Behavior Research Methods, Instruments, & Computers, 31, 565-571.

Catania, J. (1999). A comment on advancing the frontiers of sexological methods. The Journal of Sex Research, 36, 1-2.

Catania, J. A., Binson, D., Canchola, J., Pollack, L. M., Hauck, W., & Coates, T. (1996). Effects of interviewer gender, interviewer choice, and item context on responses to questions concerning sexual behavior. Public Opinion Quarterly, 60, 345-375.

Catania, J. A., Binson, D., Van Der Straten, A., & Stone, V. (1995). Methodological research on sexual behavior in the AIDS era. Annual Review of Sex Research, 6, 77-125.

Catania, J. A., McDermott, L., & Pollack, L. (1986). Questionnaire response bias and face-to-face interview sample bias in sexuality research. The Journal of Sex Research, 22, 52-72.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.

Computer Economics as cited by CyberAtlas (2000). Gay and lesbian Internet population growing. Carlsbad, CA. Retrieved July 18, 2000 from the World Wide Web: http://cyberatlas.internet.com/big_ picture/demographics/article/0,1323,5901_159001,00.html.

Couper, M. P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64, 464-494.

Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Counseling Psychology, 47, 155-164.

Etter, J. F., & Perneger, T. V. (2000). Snowball sampling by mail: Application to a survey of smokers in the general population. International Journal of Epidemiology, 29, 43-48.

Frick, A., Bachtiger, M. T., & Reips, U.-D. (2001). Financial incentives, personal information and drop-out rate in online studies. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 209-220). Lengerich, Germany: Pabst Science Publishers.

Goodman, L. A. (1961). Snowball sampling. Annals of Mathematical Statistics, 32, 148-170.

Grunwald Associates as cited by CyberAtlas (2000). 40 percent of America's kids online. Burlingame, CA. Retrieved July 18, 2000 from the World Wide Web: http://cyberatlas.internet.com/big_picture/ demographics/article/0,1323,5901_390941,00.html.

Hecht, H., Oesker, M., Kaiser, A., Civelek, H., & Stecker, T. (1999). A perception experiment with time-critical graphics animation on the World-Wide Web. Behavior Research Methods, Instruments, & Computers, 31, 439-445.

Heckarthon, D. D. (1997). Respondent-driven sampling: A new approach to the study of hidden populations. Social Problems, 44, 174-200.

Hewson, C. M., Laurent, D., & Vogel, C. M. (1996). Proper methodologies for psychological and sociological studies conducted via the Internet. Behavior Research Methods, Instruments, & Computers, 28, 186-191.

Joinson, A. N. (1998). Causes and implications of disinhibited behavior on the Net. In J. Gackenbach (Ed.), Psychology of the Internet (pp. 43-60). New York: Academic Press.

Joinson, A. (1999). Social desirability, anonymity, and Internet-based questionnaires. Behavioral Research Methods, Instruments, & Computers, 31, 433-438.

Kehoe, C. M., & Pitkow, J. (1996). Emerging trends in the WWW user population. Communications of the ACM, 39. Retrieved January 15, 2001 from the World Wide Web: http://www.cc.gatech.edu/gvu/user_surveys/papers/cacm.pdf.

Kiesler, S., Siegal, J., & McGuire, T. W. (1984). Social psychological aspects of computer mediated communication. American Psychologist, 39, 1123-1134.

Klein, J. D., & Higgens, N. (1992). Incentive contingencies, motivation, and performance in a visual learning task. Journal of Instructional Psychology, 19, 113-117.

Korn, J. H., & Hogan, K. (1992). Effect of incentives and aversiveness of treatment on willingness to participate in research. Teaching of Psychology, 19, 21-24.

Krantz, J. H., & Dalal, R. (2000). Validity of web-based psychological research. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 35-57). San Diego, CA: Academic Press.

Maughan, M. R., & Higbee, K. L. (1981). Effect of subjects' incentives for participation on estimated compliance for self and others. Psychological Reports, 49, 119-122.

McGraw, K. O., Tew, M. D., & Williams, J. E. (2000). The integrity of Web-delivered experiments: Can you trust the data? Psychological Science, 11, 508-512.

McMurdo, G. (1995). Netiquettes for networkers. Journal of Information Science, 21, 305-319.

McNemar, Q. (1942). Opinion-attitude methodology. Psychological Bulletin, 43, 289-374.

Mehta, R., & Sivadas, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of the Market Research Society, 37, 429-439.

Michalak, E., & Szabo, A. (1998). Guidelines for Internet research: An update. European Psychologist, 3, 70-75.

Mills, R. (1998). Cyber: Sexual chat on the Internet. Journal of Popular Culture, 32, 31-47.

Musch, J., & Reips, U. -D. (2000). A brief history of Web experiments. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 61-85). San Diego, CA: Academic Press.

Mustanski, B. S. (2001). Semantic heterogeneity in the definition of "having sex" for homosexuals. Manuscript in review.

National Telecommunications and Information Administration and the U.S. Department of Commerce. (1999). Falling through the Net: Defining the digital divide: A report on the telecommunications and information technology gap in America. Washington, D.C. Retrieved July 18, 2000 from the World Wide Web: http://www.digitaldivide.gov/.

National Telecommunications and Information Administration and the U.S. Department of Commerce. (2000). Falling through the Net: Towards a digital divide. Washington, D.C. Retrieved January 11, 2000 from the World Wide Web: http://www.digitaldivide.gov/.

Nielsen//NetRatings as cited by CyberAtlas. (2000). Web remains a man's world outside US. Milpitas, CA. Retrieved July 18, 2000 from the Word Wide Web: http://cyberatlas.internet.com/big_picture/demographics/ article/0,1323,5901_409541,00.html.

!OutProud!/Oasis. (1998). Internet survey of queer and questioning youth. San Rafael, CA: !OutProud!, The National Coalition for Gay, Lesbian, Bisexual, and Transgender Youth & Oasis Magazine. Retrieved March 1998 from the World Wide Web: http://www.outproud.org/.

Parks, M. R., & Floyd, K. (1996). Making friends in cyberspace. Journal of Computer Mediated Communication, 1. Retrieved July 2000 from the World Wide Web: http://www.ascusc.org/jcmc/vol1/issue4/parks.html.

Paulhus, D. L. (1988). Assessing self-deception and impression management in self-reports: The Balanced Inventory of Desirable Responding. (Manual available from the author at the Department of Psychology, University of British Columbia, Vancouver, B.C., Canada V6T 1YT).

Reips, U. -D. (2000). The Web experiment method: Advantages, disadvantages, and solutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 89-114). San Diego, CA: Academic Press.

Reips, U. -D., & Bachtiger, M. T. (2000). Are all flies drosophilae? Participant selection bias in psychological research. Manuscript in preparation.

Rheingold, H. (1993). The virtual community: Homesteading at the electronic frontier. Reading, MA: Addison-Wesley.

Roper Center at the University of Connecticut (1999a). Do you think that the increasing use of computers and the Internet in our society is increasing or reducing the level of personal privacy that people have? Retrieved July 2000 from the LEXIS-NEXIS Academic Universe database (References/Polls & Surveys).

Roper Center at the University of Connecticut (1999b). How possible is it that you might participate at some time in this kind of Internet program that exchanges benefits for information, with full explanation of how the information would be used? Retrieved July 2000 from the LEXIS-NEXIS Academic Universe database (References/Polls & Surveys).

Ross, M. W., Tikkanen, R., & Mansson, S. A. (2000). Differences between Internet samples and conventional samples of men who have sex with men: Implications for research and HIV interventions. Social Science & Medicine, 51, 749-758.

Sanders, S. A., & Reinisch, J. M. (1999). Would you say you "had sex" if ...? Journal of the American Medical Association, 281, 275-277.

Schmidt, W. C. (2000). The server-side of psychology web experiments. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 285-310). San Diego, CA: Academic Press.

Shea, V. (1994). Core rules of netiquette. Educom Review, 29, 58-62.

Shultz, D. P. (1972). Readings for experimental psychology. New York: Holt.

Tomporowski, P. D., & Simpson, R. G. (1993). Method of recruiting subjects and performance on cognitive tests. American Journal of Psychology, 106, 499-522.

Unites States Internet Council (1999). State of the Internet: USIC's report on use and threats in 1999. Washington, D.C. Retrieved July 19, 2000 from the World Wide Web: http://www.usic.org/papers/stateoftheinternet99.htm.

Welch, S. (1975). Sampling by referral in a dispersed population. Public Opinion Quarterly, 39, 237-245.

Welch, N., & Krantz, J. H. (1996). The World-Wide Web as a medium for psychoacoustical demonstrations and experiments: Experience and results. Behavioral Research Methods, Instruments, & Computers, 28, 192-196.

Williams, T. J., Pepitone, M. E., Christensen, S. E., Cooke, B. M., Huberman, A. D., Breedlove, N. J., Breedlove, T. J., Jordan, C. L., & Breedlove, S. M. (2000). Finger-length ratios and sexual orientation. Nature, 404, 455-456.

Wiseman, F., Schafer, M., & Schafer, R. (1983). An experimental test of the effects of a monetary incentive on cooperation rates and data collection costs in central-location interviewing. Journal of Marketing Research, 20, 439-442.

Zimbardo, P. G. (1969). The human choice: Individuation, reason, and order vs. deindividuation, impulse and chaos. In W. J. Arnold &. D. Levine (Eds.), Nebraska symposium on motivation (pp. 237-307). Lincoln, NE: University of Nebraska Press.

Manuscript accepted November 17, 2001

Address correspondences to Brian Mustanski, Indiana University Department of Psychology, 1101 East 10th St., Bloomington, IN 47405; e-mail: bmustans @indiana.edu.