Preventing data fabrication in telephone survey research.

Introduction

A research team at the University of Prince Edward Island, Canada, discovered that the data collection company hired to conduct telephone interviews fabricated 23% of the data sets. To prevent a recurrence, the team developed a menu of options which research administrators and others concerned with research ethics can disseminate to program directors and researchers within their organizations for application pre, during, and post data collection. Menu items relate to (i) the organizational structure of data collection companies; (ii) strategies in developing contracts with companies; (iii) operational procedures; (iv) data/record review; (v) budget; and (vi) national or international standards.

Research planners are encouraged to incorporate suggestions from the menu of options. Funders are encouraged both to require and to fund quality assurance initiatives. A call for professionalization and accreditation of data collection companies aims to address quality assurance issues in survey research.

In 2003 at the University of Prince Edward Island, Canada, the Smoke-Free Homes Research Project was jeopardized by a survey firm hired to conduct telephone interviews. The research featured a baseline population survey of 1,410 households in the first year, a subsequent social marketing intervention, and, in the second year, a post-test of another 1,410 household interviews. Challenges to data collection included strict inclusion criteria, a small population base in both intervention and control sites, and requirements to complete data collection within a one-month period each year.

The survey firm contracted to conduct the interviews delivered the stipulated number of data sets on time. During an examination of the data, in year two, inexplicable consistencies in text portions of a number of surveys raised questions about their veracity. It became evident that many interviews were manufactured by copying and pasting the whole or parts of genuine interviews to create the number specified in the contract. A re-examination of year-one data uncovered similar, but more cunningly concealed, fabricated data. In all, 23% of surveys were found to be fabricated. Fortunately, an unexpectedly large effect size meant that, even with the loss of these data, this particular study was not underpowered.

There is an emerging literature within professional research organizations (Johnson, Parker & Clements, 2001; "Methods of Interviewer Fraud Detection", 2003), in conference presentations (Caspar, 2003; Qi, 2002), and academic publications (Marshall, 2000) that concentrates on the interviewer as the site of fraudulent activity. This narrow focus can blind researchers to the possibility that fraud can be perpetrated by others in the data handling hierarchy.

This paper draws the attention of the whole research community to the reality of survey fraud originating beyond the interviewer level. Researchers need to incorporate, in survey planning, adequate procedures for the prevention and detection of telephone survey fraud. We present a menu of options citing advantages and disadvantages under six themes: (i) organizational structure of data collection companies, (ii) strategies in developing contracts with companies, (iii) operational procedures, (iv) data/record review, (v) budget, and (vi) national or international standards.

Results

I. Organizational Structure of Data Collection Organizations

Data quality assurance can be enhanced if researchers avoid contracting with companies where managers and staff are also engaged in for-profit work such as telephone promotions, sales, and customer service. Two options exist.

A. Develop In-house, University-based Survey Centres.

Advantages: University-based survey centres operate within a research culture. Research, knowledge development, and discovery are valued in their own right, and are seen as more important than generation of profit and other business motivations. The chances of someone falling into fraud are minimized. People in university-based survey centres develop familiarity with the research process, ethics review, and the procedures for reporting concerns about data fabrication.

Disadvantages: Such centres require start-up investments and can require institutional subsidization. Because of the large volume of research survey work needed to sustain such centres without institutional subsidization, they might be few in number, eliminating the possibility of locally-conducted research surveys in many universities. Managers of such centres are likely to have a cost-recovery motivation.

B. Utilize External Companies That Limit Their Work to Research Contracts. Advantages: External companies operating primarily on research contracts might be more likely to incorporate a research-sensitive perspective than companies that put their phone banks and staff to other work. Staff members are likely to develop familiarity with relevant aspects of the research process. Disadvantages: Such specialized companies are relatively few in number and tend to be located in large centres, making locally conducted research surveys impossible for some researchers. Private companies still have a strong profit motivation.

II. Strategies in Developing Contracts with Companies

In the following paragraphs, we discuss the advantages and disadvantages of nine strategies that one might use in selecting and negotiating with a research company.

A. Utilize Third-Party Security/Systems Consultation.

Researchers who are on unfamiliar ground when discussing data systems and procedural requirements with a bidder should consult with a third party systems consultant.

Advantages: This strategy brings a new perspective and skill set to the research group, adding independent verification and observation to the process.

Disadvantages: Reliance on an external consultant increases the number of contracts and relationships that need to be managed and have their costs estimated. The advice of the consultant could increase the complexity of contract negotiations with prospective companies.

B. Obtain References From a Variety of Sources, Including Research Sources.

Advantages: Researchers are better placed than are non-researchers to provide references that speak to the issues most important in the research process and are best able to comment on matters relevant to data integrity in the research enterprise.

Disadvantages: Previous clients, including researchers, might not have the information necessary to confirm that data were reported in an honest fashion. Experiences with pleasant and efficient managers can lead to a halo effect in which integrity is assumed, and reported, in references.

C. Pay for the Data Collection Organization's Time and Effort, Not for Numbers of Completed Surveys.

Advantages: This strategy helps to separate business pressures from data integrity efforts, reducing the motivation for data fabrication.

Disadvantages: The researcher is unable to budget precisely for the survey. This could result in a smaller sample size than needed when the money runs out, or necessitate paying a higher fee than expected for the full sample size. Data suppliers have little motivation to encourage efficiency in staff. Companies could be motivated to exaggerate the time and effort required to complete the work.

D. Allow Data Collection Organizations to Renegotiate Contracts If Problems in Recruitment and Completion Develop.

Advantages: Companies will be less motivated to fabricate data if time lines and compensation contain some flexibility than if they are required to abide by an initial agreement in which they substantially underestimated time and costs.

Disadvantages: Researchers select data collection organizations in part on the basis of cost, and companies could deliberately under-bid if renegotiation is easy. Researchers could be left with a smaller sample size than originally contracted for or a higher bill for data collection than expected.

E. Stepped-contracts.

Stepped-contracts provide agreement to produce a certain proportion of the calls for a certain fee, and the fee for the remaining calls is then renegotiated depending on the resources required to complete the first phase of the survey.

Advantages: Companies are motivated to perform reasonably well without great financial risk, and researchers know what their costs will be part-way through the process.

Disadvantages: Costs for the researcher are unknown until after data collection is partially completed. Changing companies if agreement cannot be reached at the renegotiation stage can be problematic or, with time-sensitive projects, entirely not feasible.

F. Divide Initial Data Collection Between Two Data Collection Organizations.

Researchers could begin with two companies in preliminary data collection, paying both for the first phase, and selecting the ongoing company on the basis of time, effort, and results.

Advantages: This strategy highlights discrepant findings between data collection organizations. It puts companies on notice regarding the importance of following agreed-upon procedures.

Disadvantages: Researchers' work in recruiting companies, training personnel, and monitoring the process is higher than with one company. Interpreting the source of discrepancies between companies could be difficult. A large sample of preliminary data might be required to make an informed assessment of the relative merit of the companies. The strategy is costly.

G. Divide Data Collection Between Two Organizations for Duration of Project.

Advantages: This strategy highlights discrepant findings and puts companies on notice regarding the importance of following agreed upon procedures. Risk is distributed across two companies.

Disadvantages: The work in recruiting companies, training personnel, and monitoring the process is greater than with one company. Interpreting the source of discrepancies between companies could be difficult.

H. Engage in Explicit Discussions and Writing Regarding Integrity, Cheating, and Skepticism.

Advantages: Holding explicit discussions about these issues, and putting them in writing, highlight their importance for data collection organizations, and notifies companies that researchers are alert to possible fraud and will test for cheating. The perceived risk of being caught is increased and this could decrease the likelihood of cheating.

Disadvantages: Such discussions might highlight, for unscrupulous company managers, the importance of care and creativity in cheating so that the possibility of detection is minimized. They could contribute to an adversarial, rather than cooperative, relationship between researchers and company.

I. Develop a Detailed Contract Template.

Templates for contracts with data collection organizations, providing a checklist of the issues to be considered in integrity assurance, could be developed by groups of researchers, by institutions, or by organizations of research administrators.

Advantages: This strategy normalizes and eases discussion of integrity issues. A contract template eases the workload on researchers addressing these issues for the first time.

Disadvantages: Templates can not fully match the particular needs of all projects.

III. Operational Procedures Eight procedural strategies follow.

A. Off-site Computer Backup.

Data are saved in two locations, one of which is inaccessible to the data contractor. Creating two copies is accomplished either through ongoing database replication or through frequent backups. Replication requires the software supporting the survey tool to be mirrored in a second location. Frequent transfer of the data to a backup site using compact discs or an internet link is a lower-cost alternative.

Advantages: The researcher has timely access to the data to directly monitor the project's progress. Minimizing the window of opportunity for data manipulation is a central strategy most effectively operationalized using database replication. Archived copies of data can be used for comparison.

Disadvantages: Data replication is a more complex solution than copying files. If the source database is manipulated post-collection then the mirror is also updated, obliterating the check and balance intended by data replication. Therefore, transaction logs are required to document changes made to the database. Maintaining a constant data link to a second site can be costly.

B. Keystroke File/Trace File (Recording All Keystrokes).

Keystroke patterns reveal date, time, exact response patterns, including changed answers and can indicate interview length. Appropriate software to capture the keystroke file is required.

Advantages: It is cumbersome to fabricate not only data but also an accompanying key-stroke file.

Disadvantages: To be interpretable, keystroke information must be replayed in the application software. Keystroke recording at least doubles the storage requirements for the data.

C. Audio Taping/Digital Recording.

Audio taping or digital recording of interviews with respondent consent and numeric indexing allows for selective or random monitoring of whole or part interviews.

Advantages: Fabricating data is difficult because a complete audio recording of the interview has to be created. Multi-line recording technology is available for call-centre based operations.

Disadvantages: The informed consent process is more complex and time consuming. For some types of research, voice recording can introduce the potential for bias. For example, respondents might be reticent to have their voices recorded and could refuse to participate, or might be more likely to give socially desirable responses. Recording systems are an added expense and, unless the information is compressed, have large storage requirements.

D. Include Open Ended Questions and Record Data in Entirety.

Construct an interview guide to include open ended responses with the expectation that interviewers record responses in their entirety.

Advantages: The inclusion of text responses increases the likelihood of detection of duplicated data.

Disadvantages: Interviews could take longer to complete. Respondent burden could be increased as interviewers seek clarification of responses.

E. Data Base Journaling.

Database journaling documents access and changes to the central data file. Creating periodic reports of this activity can reveal anomalies in data entry or application behaviour. Record keeping should be discussed as part of the contract and agreement reached on the types of records and retention time of the information to be recorded.

Advantages: Database journaling makes data tampering easier to detect.

Disadvantages: In a small operation, it can be the same person or an accomplice who reviews the logs. Storing journal files and logs increases both storage and server requirements as well as the effort required to check reports and investigate anomalies.

F. Maintain A Log File to Track Access Times and Login Information.

Advantages: This is easy to implement and the researcher is fully informed of workflow and performance issues in data collection.

Disadvantages: The supplier might not wish to report on individual employees or might not have the information in a form that is usable by the researcher.

G. Require Signature of Operator for Each Completed Interview.

Advantages: This compounds the fraud necessary for a manager/supervisor to manufacture data. It sends a message throughout the company that this is serious work.

Disadvantages: Signatures can be forged.

H. Researcher Oversees Data Collection on Site.

Advantages: With this strategy the researcher is a clear and constant presence in the data collection process.

Disadvantages: Having a researcher on site as a regular part of the data collection protocol might not be feasible. This is an extreme time commitment for researchers.

IV. Data/Record Review

A. Auditing Data Review Protocols.

Data preparation is a standard pre-analysis process where each variable is examined to ensure the codes are legitimate according to the data dictionary for the project.

Advantages: Data preparation is part of the contract. The contractor provides both the original file and the prepared version along with a list of all the changes. This allows for comparisons to be made if necessary.

Disadvantages: It is time consuming to audit the prepared data file to ensure it was not altered inappropriately.

B. Batch Analyses to Detect Anomalies in Distributions.

As data accrue, check each batch for distribution anomalies such as skew and kurtosis.

Advantages: The researcher has an early warning if the data are not conforming to statistical assumptions for distributions and can follow up to solve problems. The time required for analyses on each batch of data is minimized by using a standard syntax for the analyses.

Disadvantages: This process assumes regular data updates and that each batch of data will have a distribution similar to all others in the project. It is difficult to judge a distribution if it is based on a small number of observations and is in a new research area where little is known about the population parameters. This strategy requires particular statistical expertise on the research team.

C. Use of External Controls.

Incorporate a variety of external controls for cross validation purposes such as access to records of all phone calls and interviewer time/pay sheets.

Advantages: This strategy provides methods to cross validate specific data sets and increases the difficulty of creating undetectable, fraudulent data.

Disadvantages: This method is costly and time consuming and is useful only in situations where data fraud is strongly suspected.

D. Research Staff Conduct Call-backs to Check Veracity.

Advantages: This strategy substantially increases the likelihood of being caught if fabricating data, thereby reducing the willingness to do so.

Disadvantages: The strategy requires recording contact information from participants that might decrease their willingness to take part in the survey, especially when dealing with sensitive material. The strategy imposes an additional burden on participants. It might be difficult to re-contact some participants. Some participants might forget their original participation, or be unwilling to discuss it, leading to false-positive indications of data fraud. The strategy involves some additional cost and could complicate ethics review.

V. Budget

Secure Adequate Research Budgets for Quality Assurance.

Advantages: Funders should encourage researchers to include quality assurance considerations and costs as a routine part of research planning and budget preparation. The allocation of monies for quality assurance helps to normalize an attitude of skepticism and promote an explicit review of integrity in data collection.

Disadvantages: Funds allocated for this purpose are then not available for other purposes, thereby reducing the number and/or size of research grants. Without adequate budgets researchers might view quality assurance procedures as superfluous to the research process.

VI. National (or International) Standards

A. Develop a System of Certification/ Registration of Data Managers.

Advantages: Professionalization of the data manager role brings with it the development of codes of ethics and standards for appropriate behaviour. Certification or registration also assumes a regulatory body that can not only assess qualifications but also investigate complaints and impose sanctions for unethical conduct.

Disadvantages: Such systems are cumbersome to develop, and it is unclear what the criteria for entry into the profession might be. Self-regulating professions can be self-protective.

B. Develop a System of Accreditation of Survey Research Centres.

Advantages: Accreditation presumes meeting measurable standards with respect to quality control, including mechanisms for regular, external review of operating procedures. Formalizing such standards helps redefine the culture. The potential for losing accreditation could motivate high levels of monitoring for data integrity within the organization.

Disadvantages: Such systems are cumbersome to develop. It is unclear what organizations might make up the accrediting body. Such a system might be costly. An accreditation system is unlikely to include the level of fine review required to detect cheating. Accreditation might lead to an unwarranted sense of confidence among researchers.

C. Research Funders Establish a Set of Required Standards for Quality Control in Survey Data Collection.

Advantages: This strategy can make high quality control the baseline for funded survey research. It levels the playing field for both researchers and survey companies, leading to a common set of expectations. It legitimizes the need for attention and resources dedicated to quality control in survey research.

Disadvantages: Funders need to come together to establish an agreed-upon set of standards, and it is not clear what the mechanism for such joint action would be. Such a set of standards requires change in research practice, and such change might be resisted. Establishing a set of standards does not guarantee compliance with the standards, and stated compliance with standards does not guarantee absence of cheating.

Conclusion

Research administrators have multiple opportunities to minimize the risk of data fabrication by survey administrators. Of primary importance is the opportunity to draw researchers' attention to the problem so that investigators make informed decisions about how best to manage the risk in their particular contexts. Research administrators also can lead development of institutional decision making guides and contract templates so that researchers new to this issue are not continually reinventing the risk-management wheel. In some settings it will be appropriate for research administrators to advocate for the development of institutional infrastructure in support of survey research so that reliance on outside providers is reduced. Research administrators also have a leadership role in education and advocacy with funding organizations, so that funders are aware of the need for appropriate quality control measures when evaluating applications for support of telephone survey research, so that they treat budget requests to enable such quality control as not only legitimate but also high priority, and so that they promote national or international standards for quality control in survey data collection.

Data are useful only if authentic. Obtaining and maintaining authentic survey data requires heightened vigilance. Safeguarding data from fraud at all levels is simply another necessary step in the survey research process.

Authors' Note: This is an original article developed as a result of the authors' experience with fabricated data created by a survey company. The authors acknowledge with thanks funding support from the Canadian Institutes of Health Research, the Prince Edward Island Health Research Program, and the Sociobehavioural Cancer Research Network of the National Cancer Institute of Canada for the core study on smoke-free homes which led to the findings reported in this paper. Contact Philip Smith, PhD, Associate Professor, Psychology Department, University of Prince Edward Island, 550 University Avenue, Charlottetown, P.E.I., Canada C1A 4P3, Ph: [phone]. E-mail: [e-mail].

References

American Statistical Association. (2003, July). Request for comments regarding the statement on interviewer falsification in survey research [Electronic version]. Survey Research Methods Section Newsletter, 17, p. 6.

Caspar, R.A. (2003, August). A field interviewer verification system based on unobtrusive digitally recorded interviews. Paper presented at the 2003 Joint Statistical Meetings of the American Statistical Association, San Francisco, CA. Abstract retrieved November 7, 2003, from http://www.amstat.org/meetings/jsm/2003

Johnson, T. P., Parker, V., & Clements, C. (2001). Detection and prevention of data falsification in survey research [Electronic version]. Survey Research, 32(3), 1-2.

Marshall, E. (2000). How prevalent is fraud? That's a million-dollar question. Science, 290, 1662-1663.

Methods of interviewer fraud detection and prevention. (2003, August). Conference session at the 2003 Joint Statistical Meetings. Annual conference of the American Statistical Association. San Francisco, CA.

Qi, D. (2002, September). Tools for detection of fraud. Paper presented at a conference sponsored by the Office of Research Integrity (ORI), Office of Human Research Protections (OHRP), and the Association of American Medical Colleges (AAMC), Baltimore, MD.

Philip B. Smith

University of Prince Edward Island

Colleen R. MacQuarrie

University of Prince Edward Island and Department of Health and Social Services 

Government of Prince Edward Island

Rosemary J. Herbert

University of Prince Edward Island

David L. Cairns 

University of Prince Edward Island

Lorraine H. Begley

University of Prince Edward Island