How to prevent cheating in online surveys and experiments
Photo by Pixabay from Pexels
Prevent cheats from damaging your research by employing these simple, scientifically proven methods.
Increase the data quality of your research surveys and cognitive experiments by reducing the rate of cheating.
This article explains the psychology of cheating, why cheating occurs, and provides practical, easy-to-apply techniques to help prevent cheating.
Contents
Cheating
Cheating by participants in online studies can take on many forms. For example, a participant could make hand-written notes during the course of a memory task, or use a search engine on the web to research responses in knowledge-based surveys. Whatever the case, cheating reduces the quality of your data. Prevention, detection and removal of bad data is vital for good quality results.
There is relatively little research on reducing the rates of cheating in online studies. Thus, the methods discussed in this article mostly derive from a small number of published studies.
How to prevent cheating
Data quality can be seriously degraded by participants who cheat. Preventing cheating in the first place is your number one priority where quality of data is concerned. Ultimately, prevention means better data and less cleanup afterwards.Warnings and commitments
Simply telling participants not to cheat and warning them of the consequences if they choose to do so is enough to reduce the incidence of cheating. Rather than warning participants in a consent form, it's more effective to use a separate, dedicated screen. Warning participants that fraudulent activity will be reported to authorities may act as a powerful deterrent. However, it may also prevent legitimate participants from entering the study and could be seen as unethical and punitive (Teitcher et al., 2015).
A between-subjects experiment with random allocation to either an online group or an in-lab group conducted by Clifford and Jerit (2014) sampled students' political knowledge. The results from this study showed that the political knowledge amongst students in the online group was significantly better than those in the in-lab group, leading the researchers to suspect a higher rate of cheating in the online group.
Among the student samples, rates of self-reported cheating are relatively high, ranging from 24% to 41%
These results inspired a second study (Clifford & Jerit, 2016), consisting of 4 experiments, to research rates of self-reported cheating in an online, between-groups experiment consisting of students and participants from Amazon's Mechanical Turk (MTurk). Participants who self-reported cheating spent significantly longer answering knowledge-based questions when compared to those who didn't self-report cheating. This, together with the effect size observed, points to the use of outside assistance to answer those questions.
A control (no technique applied) and two prevention techniques were studied (direct request and commitment) resulting in the rates of cheating shown in the table below. Note: the direct request method was not applied to participants from the MTurk group.
| Control (no prevention applied) | Direct request | Commitment | |
|---|---|---|---|
| Student | 41% | 24% | 9.5% |
| MTurk | 6.6% | N/A | 1.5% |
As shown in the table above, Clifford and Jerit (2016) found that compelling a participant to commit to not cheating (with a yes or no response) was much more effective than warnings for reducing rates of cheating.
Attaining commitment to not engage in cheating can be achieved with a multiple-choice question (Figure 1).
Trust values and opportunity cost
Most participant recruitment services and crowdsourcing services provide trust values for each participant which you can use as eligibility criteria for your study (Peer, Vosgerau, & Acquisti, 2013). Additionally, participants from recruitment services are already registered and vetted, so the chances of cheating and multiple submissions are reduced. Historically, cheating rates have been shown to be very low amongst paid participants recruited from these services (Berinsky, Huber, & Lenz, 2012; Chandler & Paolacci, 2017; Germine et al., 2012), however there is some conflicting evidence showing much higher rates of cheating (Motta, Callahan, & Smith, 2017). It is suspected that opportunity cost is a very strong deterrent for participants from these types of services. Simply put, it's too expensive for participants to cheat (where they believe they do not have to) because they are paid per study and the longer it takes, effectively, the less they earn.
How to detect cheating
It's unlikely that you'll be able to prevent cheating in all cases, but you can apply some methods for detecting suspicious cases and then more closely scrutinize their data.
Self-reporting
Allowing participants to self-report cheating has shown to provide an accurate aggregate measure of cheating within a study (Clifford & Jerit 2016). Self-reporting can consist of something as simple as a multiple-choice question, as shown in (Figure 2).
Timing checks
In knowledge-based tasks, if a participant has taken longer than expected to answer a question, it could be that they're researching the answers rather than supplying responses from their current knowledge. Timing checks can therefore be useful in identifying this type of cheating in the majority of tasks. However, such checks are not useful for detecting cheating in prospective memory tasks or any other task where taking notes during the task could boost performance.
Difficult and obscure questions
Motta et al. (2017) tested a detection method that involved asking participants extremely difficult and obscure knowledge-based questions, that few (if any) participants would be expected to know. All correct responses were coded as cheating. For example, in a survey testing political knowledge, the researchers asked the following difficult question:
In what year was the Alaskan Purchase Treaty signed?
Motta et al. (2017) concluded that 25% of participants (sourced from Amazon's Mechanical Turk) were found to have cheated (control); however, when given instructions to not look up answers, only 5% of participants were found to have cheated. The same study was conducted with participants recruited from SurveyMonkey with a cheating rate of 11% (control) and 8% when given instructions to not look up answers.
A somewhat surprising finding in a subsequent experiment by Motta et al. (2017), was that rates of cheating on phones and tablet computers (7%) were lower than rates on desktop and laptop computers (10%). The researchers suggest that researching answers on mobile devices requires a much greater cognitive load because it requires a large increase in workload, compared to the same task on a desktop or laptop computer. Therefore, when it comes to cheating on mobile devices, participants are more likely to satisfice when considering cheating.
Why cheat?
There are two main reasons why participants cheat on surveys and experiments (Booth-Kewley, Rosenfeld, & Edwards, 1992):
- Impression management: the deliberate tendency to over-report desirable behaviors and under-report undesirable ones.
- Self-deceptive enhancement: the tendency to give honestly believed but overly positive reports about oneself.
Impression management
Generally, impression management tends to present little problem in online experiments and surveys, particularly where participants are anonymous.
Self-deceptive enhancement
Self-deceptive enhancement appears to be common in some studies (Clifford & Jerit, 2014) and is an important motivational factor for cheating in online studies because of opportunity and easy access to information online (Kewley, Larson, & Miyoshi, 2007). According to Sparrow, Liu, and Wegner (2011), the internet allows people to delegate their memory of facts and information to the web without any conscious awareness of it taking place. Having seen, or knowing where to find such facts and information leads some participants to believe they actually possess such knowledge and are unable to tell the difference between information online and that which is personally known. In sum, some participants choose to "refresh their memory" by searching for answers online and do not recognize this as an act of cheating.
Conclusion
- Setting, personality traits, motivation, and opportunity are all contributing factors in cheating in online studies.
- Compelling participants not to cheat by asking them for a commitment appears to be significantly more effective than simply directing participants not to cheat.
- Self-reporting of cheating is recommended as an accurate way to determine the aggregate rate of cheating behavior within a study.
- Timing checks are a decent proxy for cheating behavior in knowledge-based tasks.
- Correct answers to difficult and obscure questions are a decent proxy for cheating behavior in knowledge-based tasks.
- Opportunity cost is believed to greatly decrease the rate of cheating for participants recruited via participant recruitment services and crowdsourcing platforms.
- Self-deceptive enhancement is thought to be the greatest contributor to rates of cheating in online studies, particularly where opportunity cost is negligible. Timing checks can be a useful proxy in establishing whether a participant engaged in self-deceptive enhancement.
References
- Berinsky, A., Huber, G., & Lenz, G. (2012). Using Mechanical Turk as a subject recruitment tool for experimental research. Political Analysis 20(3), 351–368.
- Booth-Kewley, S., Rosenfeld, P., & Edwards, J. (1992). Impression management and self-deceptive enhancement among Hispanic and non-Hispanic White navy recruits. The Journal Of Social Psychology, 132(3), 323-329. doi: 10.1080/00224545.1992.9924707
- Chandler, J., & Paolacci, G. (2017). Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8(5), 500–508. doi: 10.1177/1948550617698203
- Clifford, S., & Jerit, J. (2014). Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1(2), 120–131. doi: 10.1017/xps.2014.5
- Clifford, S., & Jerit, J. (2016). Cheating on political knowledge questions in online surveys. Public Opinion Quarterly, 80(4), 858–887. doi: 10.1093/poq/nfw030
- Germine, L., Nakayama, K., Duchaine, B., Chabris, C., Chatterjee, G., & Wilmer, J. (2012). Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review, 19(5), 847–857. doi: 10.3758/s13423-012-0296-9
- Kewley, S., Larson, G., & Miyoshi, D. (2007). Social desirability effects on computerized and paper-and-pencil questionnaires. Computers in Human Behavior 23(4): 463–477. doi: 10.1016/j.chb.2004.10.020
- Motta, M., Callaghan, T., & Smith, B. (2017). Looking for Answers: Identifying Search Behavior and Improving Knowledge-Based Data Quality in Online Surveys. International Journal Of Public Opinion Research, 29(4). doi: 10.1093/ijpor/edw027
- Peer, E., Vosgerau, J., & Acquisti, A. (2013). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023-1031. doi: 10.3758/s13428-013-0434-y
- Psychstudio (2019). Psychstudio (Version 2019) [Computer software]. Australia. Retrieved from https://www.psychstudio.com
- Sparrow, B., Lui, J., & Wegner, D. (2011). Google effects on memory: Cognitive consequences of having information at our fingertips. Science 333(6043) 776–78. doi: 10.1126/science.1207745
- Teitcher, J., Bockting, W., Bauermeister, J., Hoefer, C., Miner, M., & Klitzman, R. (2015). Detecting, preventing, and responding to “fraudsters” in Internet research: Ethics and tradeoffs. The Journal of Law, Medicine & Ethics, 43(1), 116–133. doi: 10.1111/jlme.12200