How to conduct questionnaire surveys.
Designing, organizing, and conducting a questionnaire survey is like establishing and running a business. Although these activities require technical knowledge and skill, administrative ability, and specific experience or training, the general impression is that, with a little luck, anyone can do them well. Thus, many people assume that starting a business requires no special skills beyond a willingness to try. The annual failure of thousands of new small businesses demonstrates the naivety of this assumption. Similarly, the findings from thousands of questionnaire surveys that are unreliable or misleading prove that conducting a viable, reliable questionnaire survey is not as easy as it might seem.
Contrary to common assumptions, a questionnaire is more than a list of questions. Rather, it is a scientific instrument for measuring and collecting particular kinds of data. And, like all other scientific instruments, a questionnaire must be designed in accordance with particular specifications and tailored to the specific aims of the surveyor. Only by careful "soup-to-nuts" planning, construction, and implementation can reliance be placed upon the results of the questionnaire survey. To assist anyone planning to conduct a questionnaire survey, this article provides a comprehensive checklist of items to consider in questionnaire development (figure 1). These items take the surveyor from the initial survey planning stages through instrument design and development and on to actual survey implementation. By following the information and suggestions provided here, surveyors can use a systematic methodology to collect valid data and achieve reliable results.
Planning the Survey
What question or questions are to be answered by the survey?
Before the survey can be undertaken, the surveyor must be able to present its objective clearly and specifically. What is the problem that the survey will address? Typically, the general nature of this problem has been determined already. However, the surveyor must further define the specific problem that is to be studied objectively and scientifically in the survey.
Next, the surveyor must determine the facts needed to answer the problem statement. What are the specific questions to be answered by the survey? What data need to be collected? Collecting information for its own sake rarely justifies the effort required to assemble and present it. Therefore, the survey should not pose questions whose answers are interesting but unessential to the main thrust of the investigation. In making these determinations, the surveyor should keep in mind the survey's end user - the agency or individuals who will use the survey findings. By keeping these potential consumers in mind, the surveyor is more likely to determine those questions that will be most meaningful and useful.
How and by whom will the results be used?
This question is related to the previous one. In developing the survey plan, the surveyor must keep in mind the characteristics and information needs of the survey's eventual consumers. While it is not possible to foresee all the misuses to which survey statistics may be put, many abuses can be prevented if the surveyor is aware of the consumers' needs.
How large an effort is needed?
Depending on the nature of the problem to be addressed and the amount of information already available on that topic, the surveyor may want to use the survey as a small preliminary or pilot study to form the basis for recommended subsequent research.
This option, however, is rarely available: more often the problem needs to be addressed in full immediately. In these cases, the surveyor should have some control or input relative to planning the survey's required funding and time. The surveyor should ensure that sufficient funds will be available not only for the collection of the necessary data but for their tabulation, analysis, and presentation. Similarly, the surveyor should ensure that sufficient time is allotted for questionnaire development, distribution, collection, followup, and for resolving any problems that may arise. If either the survey budget or schedule is under-estimated, the survey is not likely to be successful.
Is the desired information already available?
No survey should be undertaken until the surveyor is satisfied that the required facts have not already been collected. Often, a careful search will show that some of the desired information is already published or on file. However, if the available facts are inconclusive, outdated, or otherwise insufficient, the surveyor can profit by avoiding the mistakes of the earlier survey or by expanding upon those aspects that the original did not emphasize.
How will information be obtained?
Once the problem to be investigated has been defined and the decision to conduct a survey finalized, the surveyor should next decide what survey method should be used to obtain the desired information from the respondents. Three general survey procedures are available:
* Personal (face-to-face) interview.
* Telephone interview.
* Self-administered questionnaires.
The advantages and disadvantages of these procedures are shown in figure 2.
What kind of sampling is appropriate?
Although it is sometimes possible to survey an entire population, generally this technique would result in an unwieldy amount of data. To limit the population surveyed, a sample is drawn to reflect the characteristics of the total population. By using a carefully drawn sample, the surveyor is assured that potential respondents have been selected in a standard, scientific manner.
Many surveyors assume that a sample can be drawn unsystematically - e.g., by accident or self-selection - and thereby yield a random sample. This assumption is incorrect: unsystematic sampling may be biased at worst or unrepresentative at best. Because many surveyors fail to decide what population they are sampling, they have no way of knowing if they are exposing all of the population to the sampling process. A similar problem exists when the source or list from which the sample is chosen does not represent the population. Therefore, the surveyor must first define the survey population and then select a method or methods of sampling it. Following are descriptions of the most commonly used sampling types:
* Random sampling.
* Stratified sampling.
* Purposive sampling.
By using two or more of these sampling techniques, a survey can capitalize on their combined advantages and counterbalance their disadvantages. This method is called double or mixed sampling.
Random Sampling. Random sampling assures each individual or element in the population the same chance of being chosen for inclusion in the survey. If the sample is chosen at random and is sufficiently large, it will represent all the groups in the population in approximately the correct proportions. One advantage of random sampling is no advance knowledge of the characteristics of the population is necessary. A disadvantage of the technique is that a low percentage of returns could result in an accidental sample.
Stratified Sampling. Stratified sampling is frequently recommended as the most efficient procedure for ensuring representativeness. The procedure calls for dividing the population into two or more strata and then drawing a random sample from each stratum. Stratified sampling may be proportional or disproportional depending on the number of samples selected within each stratum. In proportional stratified sampling, samples are drawn from each stratum in the same proportion as they occur in the population. Consequently, the only difference between proportional sampling and random sampling is that the former ensures that the right proportion is obtained from each stratum. In disproportional stratified sampling, an equal number of samples is secured from each stratum, thereby facilitating reliable comparison of the different strata. The goal of obtaining an equal number of samples, however, is very difficult to achieve in actual practice.
Purposive Sampling. The objective of purposive sampling is to make the sample look like the population. There are several ways of doing this:
* Select the sample so its averages
ages are the same as the population
averages for known characteristics
to the phenomenon being
* Select the sample so its frequency
distribution of each relevant
the population's frequency distribution.
* Select the sample so it will contain
the same proportions as
the population's at the upper
and lower ends of the frequency
* Select the sample so that it's
variability (e.g., standard deviation)
is the same as the population
One advantage of purposive sampling is it may be cheaper than other sampling techniques. A major disadvantage is that much must be known about the population before sample selection.
How big a sample is needed?
The optimum sample size ensures efficiency, representativeness, and reliability. It should provide the most useful information with the required level of reliability at the minimum cost. The sample should be large enough to yield statistically representative and significant results in all key proposed tabulations and to avoid intolerable sample error. It should not be so large, however, as to waste funds, delay the project, or achieve a needlessly high level of precision.
When trying to determine the optimum size of the sample, the surveyor should follow the guidance provided below. In general, the surveyor should note that it is sound practice to err on the side of too large a sample rather than too small.
To determine optimum sample size, the surveyor should perform the following.
Secure preliminary information about the population. If the sample will be used to make a percentage estimate of the population, a preliminary estimate of the approximate size of this percentage is needed. On the other hand, if the purpose is to predict the average of the population, the surveyor needs an estimate of the standard deviation. Fortunately, both these estimates can be rough and still prove useful.
Determine the required precision of the prediction. This determination depends upon the error permissible in the estimate and the degree of confidence that the estimate will fall within the permissible error.
Calculate the required sample size. For estimating percentages, use
n = pq[z.sup.2]/[t.sup.2] where p = the preliminary estimate
of the percentage q = 100 - p z = the number of standard
error units (from a normal
probability table) T = the required precision or
tolerance Example: Suppose we are permitted a [+ or -] 5-percent error in estimating from a population the percentage of engineers with 10 years or more of experience. Assume we will be satisfied with a 90-percent degree of confidence that the estimate will fall within the 5-percent tolerance. We have reason to believe that the percentage of engineers with 10 years or more of experience is somewhere around 75 percent. Substituting in the formula above, we have:
n = (75)(25)[(1.64).sup.2] /[(5).sup.2]
n = 202
n = (75)(25)[(1.64).sup.2]/[5.sup.2]
Thus, we need to take a random sample large enough to assure responses from about 200 engineers if we want a 90-percent probability that the true percentage is within [+ or -] 5-percent.
For estimating averages, use [Mathematical Expression Omitted] where
[sigma] = the estimated standard
z = the number of standard
T = the required precision or
tolerance Example: Suppose we want to estimate the average design strength of concrete pavement mixes within 100 lb per square in (psi) (7.0 kg/[cm.sup.2]). Let us assume we will be satisfied with a 95-percent degree of confidence that our estimate will fall within the 100-psi (7.0 kg/[cm.sup.2]) tolerance.
If we use 300 psi (21.1 kg/[cm.sup.2]) as the estimated standard deviation of concrete pavement mix design strength, we can substitute in the formula above: [Mathematical Expression Omitted]
n = 35
Thus, we need to take a random sample large enough to assure 35 responses regarding concrete pavement mix design strength if we want a 95-percent probability of estimating the average design strength within [+ or -] 100 psi (7.0 kg/[cm.sup.2]).
If the preliminary estimate of either the percentage or standard deviation proves to have been too large, the computed sample size also will have been larger than necessary and therefore conservative; the preliminary estimate may also come out too small, with the opposite result. To help set sample size, the surveyor should try various estimates in the formula to see what effect each has on the computed sample size.
Construction of the
Is there a clear understanding of the topic to be addressed?
If the questionnaire is to be meaningful to others, the surveyor needs to have a good working knowledge of the topic to be addressed. If the surveyor does not fully understand the topic, the questions may be vague or ambiguous, complex issues may be oversimplified, insufficient space may be provided for responses, etc. In such cases, respondents may show their frustration by either rushing through the questionnaire, providing only superficial answers or not responding at all.
Are needed definitions provided?
Related to understanding the survey topic is the concept of understanding the respondents. The surveyor must: * Understand the respondents'
language, and use it correctly.
* Not talk down to the respondents.
* Not assume too quickly that respondents
will understand the
question in the intended frame
* Ensure that the questions have
the same meaning for all respondents.
Additionally, definitions of key terms used in the questionnaire should be provided. There should not be too many definitions, nor should the definitions be lengthy or complicated. The definitions may appear together in a section above the questions themselves; if only one or two definitions are to be given, it may be more appropriate to include them in notes following the respective question. The objective is to get the respondents to read and understand each definition.
How long should the questionnaire be?
The percentage of questionnaires returned decreases as the number of pages increases from 1 to 10. (Interestingly, there does not seem to be a significant difference in the percentage of returns as pages increase from 10 to 35.) Although evidence suggests that with the proper incentives and a carefully pretested form, certain groups will respond thoroughly to a very long questionnaire. However, the best approach is to keep the form as short as possible to obtain the needed information.
To keep the questionnaire's length down, the surveyor should evaluate each proposed question separately. Only those questions bearing directly on the problem should be included; answers to question that can be secured from other sources may be deleted unless absolutely necessary as a check.
Are the right type(s) of questions being asked?
There is no generally accepted type of question that can be universally recommended for all questionnaires. Rather the type(s) of questions selected for use in a given questionnaire should depend primarily on the anticipated tabulation plan (i.e., how will the data be grouped in the tables?). To determine the appropriate type(s) of questions to use, the surveyor should both consider the tabulation plan and the strengths and limitations of the various types of questions as presented below.
Open-End Questions. Open-end questions give respondents free latitude in making their responses. An example of such a question is "What is the principal reason for your dislike of product A?" A drawback to their use is that open-end questions may yield such a variety of responses that the task of condensing and analyzing them is both time consuming and difficult to handle statistically. Open-end questions are often used in the exploratory phases of a study when the area covered by the question is still not well understood.
Dichotomous Questions. These questions elicit either/or responses (yes/no, true/false, good/bad, etc.) They usually also allow for a third response such as "don't know," "neither," etc. Surveryors, however have found that if the noncommittal third alternative is not given, relatively few respondents will fail to make a choice between the two options presented, thereby "forcing" some respondents to take definite stand when actually they would not do so otherwise. The chief advantage of the dichotomous question is its simplicity. One of its limitations is that a slight misunderstanding of the question's meaning may result in a complete reversal of response. (3)
Multiple Choice Questions. Multiple choice questions allow respondents to choose from among several possible answers. The list of alternatives must be complete enough to cover all possible answers. Multiple choice question results can be tabulated easily and interpreted with greater precision than can answers to open-end questions. Multiple choice questions may take any of several forms; arguably the most popular of these are checklists and rating scales.
Checklists. Checklist are multiple choice questions consisting of a statement of the problem or question followed by a list of three or more possible answers. Checklists should be made up only after some preliminary survey or pretest has determined the types of replies that may be expected; this ensures that most answers will not fall in the "other" category that is usually provided with a limited list. The advantage of the checklist is that it serves to remind respondents of their various options.
Rating Scales. Rating scales allow respondents to choose among various degrees of opinion. The scale may be numerical (e.g., ranging from 1 to 5) or verbal (e.g., ranging from "poor" to "excellent"). Most dichotomous questions can be turned into rating scale questions simply by introducing degrees of opinion to the response options. After doing so, the extreme responses can be sorted out for use in comparative analyses where a high statistical significance level is desired. One problem with rating scale questions, however, is there is often a tendency for the respondent to choose a moderate rating. Ranking of Items. In ranking of items questions, the respondent is asked to arrange a list of words, phrases, statements, pictures, etc., in some order. The ranking adds a dimension not found in checklists. However, a limitation of these questions is the fact that the size of the intervals between rankings is usually not the same: the difference between the first and second choices may be much greater than the difference between the second and third choices. Another limitation is the relatively small number of items that may be included. It has been suggested that as few as three choices give the best results; the optimum number probably varies with the subject matter. (3)
Are questions properly worded?
Asking questions to gain desired responses is more difficult than it may seem. The greatest danger in wording questions lies in taking too much for granted regarding language and terminology, reader comprehension and background, etc. Surveyors must develop a critical attitude toward their questions and should carefully analyze the wording of each. The critique is best done during pretest of the questionnaire. Every objection that may be raised about the phrasing should be carefully studied and answered. (4)
Following are some suggestions for wording questions:
* Use simple words and correct punctuation
* Use underline or boldface type for emphasis.
* Make questions concise and unambiguous.
* Formulate questions to yield exactly the information desired (e.g., specify the units for the answer) and avoid leading questions, i.e., questions worded in such a way as to suggest the answers.
* Avoid words such as "always" or "any," "about" or "near," "possible," and "should." When using "you" or "your," make it clear whether the second person singular or plural is intended.
* Cover only one point per question. If there are two or more ideas in the question, it will be confusing to determine to which thought the answer applies.
* Allow for all possible responses, i.e., multiple choice answers should be mutually exclusive and complementary and provide for "don't know" and "other" answers.
* Keep asking "What am I assuming or taking for granted by this question?"
Are questions in logical order?
To avoid confusion and misunderstanding, questions should be arranged logically. The questionnaire should, for example, open with a question that can be easily answered, preferably with a simple yes or no. If a difficult question is asked at the beginning, respondents may refuse to continue with the questionnaire.
After this important opening question, there must be an easy-to-follow logic, an established order, to the form. Order should be considered at three presentation levels:
* Topics covered by the questionnaire.
* Questions under each topic.
* Response alternatives to a single question.
The surveyor should note that once a train of thought has been established, breaking that sequence can lead to poor results. Thus, a totally unrelated question, suddenly, asked, is not desirable.
Is the questionnaire attractive?
An attractive questionnaire is a strong selling point for respondent cooperation. Attractiveness can be accomplished by the appropriate use of colors, pictorial materials, spacing, type, arrangement, etc. Research has shown that yellow paper - followed closely by pink - has the highest percentage of returns; dark-colored questionnaires usually are not effective. (3) The questionnaire should look easy to complete. This goal can be met by requiring very little writing of the respondent and/or arranging the various items so the questionnaire does not appear crowded.
Sending the Questionnaire
Has the questionnaire been pretested?
Regardless of how carefully the questionnaire has been worded, it should not be assumed that it will work well until it has been tested under field conditions. A pretest can be used to check the wording and sequence of questions, questionnaire length, clarity of instructions, effectiveness of cover letter, and other related items. Pretests also enable the surveyor to turn open-ended questions into multiple choice ones.
A pretest is normally done by furnishing the questionnaire to typical respondents, reviewing their responses, and interviewing them to get their reactions, suggestions, and other comments. Based on this feedback, the questionnaire is then revised as appropriate. Especially difficult questionnaires may require several revisions.
Another way to pretest the questionnaire is by developing several different drafts and simultaneously sending a different draft to each of several small samples of respondents. Such a pretest will provide comparative information on relative rates of return, data completeness and appropriateness, data suitability for tabulation, etc.
Is the cover letter effective?
A letter of transmittal soliciting respondent cooperation should accompany the questionnaire. This letter should:
* Be on official letterhead.
* Explain the purpose of the survey and the uses to be made of the data.
* Provide instructions on how to fill out the form (if such instructions are not printed on it).
* Thank the respondent for cooperating.
Customized, personal touches to the letter can be effective in motivating potential respondents. Examples include a hand-written postscript or personal signature. Another possible "motivator" is including self-addressed stamped or franked envelopes. Finally, promising to furnish survey results to respondents can also be effective. Of course, such a promise must be fulfilled in a timely manner.
When should the questionnaire be sent?
The surveyor must determine the most opportune time for sending out the questionnaire. This choice may be based on several factors, since the season of the year, the month, week, or time of the week during which a questionnaire should arrive in order to have the best chance of being filled out will vary with the type of respondent. (For example, a survey of construction engineers might best be done in the winter months when work is not at its peak.) The surveyor should also take into account the timing of the mailing vis-a-vis periods such as holidays, extremely hot weather, and important national events.
When and how to followup?
Followup of some kind is usually required to ensure an adequate rate of response. (3) The first followup might take the form of a short note or postcard reminding the recipient of the overdue questionnaire. A tactful suggestion may be included to disregard the reminder if the reply is on the way. The second reminder may include a copy of the questionnaire since the respondent may have misplaced the original. This second followup may also take the form of a telephone call.
The first followup activity should be undertaken when the daily returns have dwindled down to a trickle. Subsequent followups, if needed, should be spaced closer and closer together.
The acceptable percentage of responses depends on such variables as type of respondent, size of the questionnaire, subject matter, etc. However, it is not unrealistic to expect a 75-percent or greater rate of response for most questionnaires.
This article is intended to provide guidance for the development of generic questionnaires. While the examples provided are engineering- or highway-related, the discussion may be equally applicable to questionnaires in other fields.
There is more to developing good questionnaires than just wording good questions. Weeks of planning and exploratory work are often needed. Thus, it is important that the surveyor schedule sufficient time. Successful questionnaire development includes identifying the specific problem, understanding the subject matter, and knowing the respondents. Appropriate survey method, sample size, type of sampling, and type of questions have to be given adequate consideration. The use of pretest is probably the greatest help in devising the actual wording of questions. The surveyor should also think well ahead of the data analysis phase. Because each survey tends to present its own problems, no amount of advice or guidance can substitute for thoughtful care, common sense, and good organization.
 J.C. Stamp. Criticism and Other Addresses, Books for Libraries Press, Salem, NH, 1966.
 A.N. Oppenheim. Questionnaire Design and Attitude Measurement, Basic Books, Inc., New York, 1966.
 M.B. Parten. Surveys, Polls, and Samples: Practical Procedures, Harper and Brothers Publishers, New York, 1950.
 S.L. Payne. The Art of Asking Questions, Princeton University Press, Princeton, 1973.
 S. Dutka, L.R. Frankel, and J. Roshwalb. Audits and Surveys, Audits and Surveys, Inc., New York, 1982.
 Office of Management and Budget, "Memorandum on Reporting Burden Reduction Program to Heads of Executive Departments," February 17, 1977, Washington, DC.
PHOTO : Figure 1. - Questions development checklist.
PHOTO : Figure 2. - Advantages and disadvantages of three popular survey procedures.
Peter A. Kopac is a research highway engineer in the Pavements Division, Office of Engineering and Highway Operations Research and Development, Federal Highway Administration (FHWA). He is a program manager for the Nationally Coordinated Program, "Highway Maintenance." Since joining the FHWA in 1977, Mr. Kopac has worked in the areas of statistical quality control, data collection and analysis, and performance-related specifications. His responsibilities have included management and design of numerous research studies that relied on questionnaire surveys.
|Printer friendly Cite/link Email Feedback|
|Author:||Kopac, Peter A.|
|Date:||Jun 1, 1991|
|Previous Article:||Color recognition of retroreflective traffic signs under various lighting conditions.|
|Next Article:||Technology transfers and its tools.|