Photo by Pixabay from Pexels
Prevent cheats from damaging your research by employing these simple, scientifically proven methods.
Increase the data quality of your research surveys and cognitive experiments by reducing the rate of cheating.
This article explains the psychology of cheating, why cheating occurs, and provides practical, easy-to-apply techniques to help prevent cheating.
Cheating by participants in online studies can take on many forms. For example, a participant could make hand-written notes during the course of a memory task, or use a search engine on the web to research responses in knowledge-based surveys. Whatever the case, cheating reduces the quality of your data. Prevention, detection and removal of bad data is vital for good quality results.
There is relatively little research on reducing the rates of cheating in online studies. Thus, the methods discussed in this article mostly derive from a small number of published studies.
Simply telling participants not to cheat and warning them of the consequences if they choose to do so is enough to reduce the incidence of cheating. Rather than warning participants in a consent form, it's more effective to use a separate, dedicated screen. Warning participants that fraudulent activity will be reported to authorities may act as a powerful deterrent. However, it may also prevent legitimate participants from entering the study and could be seen as unethical and punitive (Teitcher et al., 2015).
A between-subjects experiment with random allocation to either an online group or an in-lab group conducted by Clifford and Jerit (2014) sampled students' political knowledge. The results from this study showed that the political knowledge amongst students in the online group was significantly better than those in the in-lab group, leading the researchers to suspect a higher rate of cheating in the online group.
Among the student samples, rates of self-reported cheating are relatively high, ranging from 24% to 41%
These results inspired a second study (Clifford & Jerit, 2016), consisting of 4 experiments, to research rates of self-reported cheating in an online, between-groups experiment consisting of students and participants from Amazon's Mechanical Turk (MTurk). Participants who self-reported cheating spent significantly longer answering knowledge-based questions when compared to those who didn't self-report cheating. This, together with the effect size observed, points to the use of outside assistance to answer those questions.
A control (no technique applied) and two prevention techniques were studied (direct request and commitment) resulting in the rates of cheating shown in the table below. Note: the direct request method was not applied to participants from the MTurk group.
|Control (no prevention applied)||Direct request||Commitment|
As shown in the table above, Clifford and Jerit (2016) found that compelling a participant to commit to not cheating (with a yes or no response) was much more effective than warnings for reducing rates of cheating.
Attaining commitment to not engage in cheating can be achieved with a multiple-choice question (Figure 1).
Most participant recruitment services and crowdsourcing services provide trust values for each participant which you can use as eligibility criteria for your study (Peer, Vosgerau, & Acquisti, 2013). Additionally, participants from recruitment services are already registered and vetted, so the chances of cheating and multiple submissions are reduced. Historically, cheating rates have been shown to be very low amongst paid participants recruited from these services (Berinsky, Huber, & Lenz, 2012; Chandler & Paolacci, 2017; Germine et al., 2012), however there is some conflicting evidence showing much higher rates of cheating (Motta, Callahan, & Smith, 2017). It is suspected that opportunity cost is a very strong deterrent for participants from these types of services. Simply put, it's too expensive for participants to cheat (where they believe they do not have to) because they are paid per study and the longer it takes, effectively, the less they earn.
It's unlikely that you'll be able to prevent cheating in all cases, but you can apply some methods for detecting suspicious cases and then more closely scrutinize their data.
Allowing participants to self-report cheating has shown to provide an accurate aggregate measure of cheating within a study (Clifford & Jerit 2016). Self-reporting can consist of something as simple as a multiple-choice question, as shown in (Figure 2).
In knowledge-based tasks, if a participant has taken longer than expected to answer a question, it could be that they're researching the answers rather than supplying responses from their current knowledge. Timing checks can therefore be useful in identifying this type of cheating in the majority of tasks. However, such checks are not useful for detecting cheating in prospective memory tasks or any other task where taking notes during the task could boost performance.
Motta et al. (2017) tested a detection method that involved asking participants extremely difficult and obscure knowledge-based questions, that few (if any) participants would be expected to know. All correct responses were coded as cheating. For example, in a survey testing political knowledge, the researchers asked the following difficult question:
In what year was the Alaskan Purchase Treaty signed?
Motta et al. (2017) concluded that 25% of participants (sourced from Amazon's Mechanical Turk) were found to have cheated (control); however, when given instructions to not look up answers, only 5% of participants were found to have cheated. The same study was conducted with participants recruited from SurveyMonkey with a cheating rate of 11% (control) and 8% when given instructions to not look up answers.
A somewhat surprising finding in a subsequent experiment by Motta et al. (2017), was that rates of cheating on phones and tablet computers (7%) were lower than rates on desktop and laptop computers (10%). The researchers suggest that researching answers on mobile devices requires a much greater cognitive load because it requires a large increase in workload, compared to the same task on a desktop or laptop computer. Therefore, when it comes to cheating on mobile devices, participants are more likely to satisfice when considering cheating.
There are two main reasons why participants cheat on surveys and experiments (Booth-Kewley, Rosenfeld, & Edwards, 1992):
Generally, impression management tends to present little problem in online experiments and surveys, particularly where participants are anonymous.
Self-deceptive enhancement appears to be common in some studies (Clifford & Jerit, 2014) and is an important motivational factor for cheating in online studies because of opportunity and easy access to information online (Kewley, Larson, & Miyoshi, 2007). According to Sparrow, Liu, and Wegner (2011), the internet allows people to delegate their memory of facts and information to the web without any conscious awareness of it taking place. Having seen, or knowing where to find such facts and information leads some participants to believe they actually possess such knowledge and are unable to tell the difference between information online and that which is personally known. In sum, some participants choose to "refresh their memory" by searching for answers online and do not recognize this as an act of cheating.
Conduct simple psychology tests and surveys, or complex factorial experiments. Increase your sample size and automate your data collection with experiment software that does the programming for you.
Behavioral experiments. No code.