A Comparison of Different Data Quality Control Methods through the Magnitude Effect in Delay Discounting: A Practical Guide for MTurk Research

Amazon Mechanical Turk (MTurk) is one of the most popular crowdsourcing platforms for online research. Although the recent MTurk data quality crisis has raised serious concerns regarding its utility, this issue can be mitigated by including quality control methods. To date, only a few studies have compared the effectiveness of the methods in preserving an experimental effect. To provide empirical guidance on which quality control methods to use for MTurk research, this study compared different methods on their effectiveness in preserving the magnitude effect in delay discounting, a robust finding in the behavioral economics literature. Two groups of MTurkers, whose HIT approval rate (%) was between 95 and 98 and 99 and 100, were recruited in October 2021. They completed a delay discounting task evaluating the magnitude effect and questions from different quality control methods. The results showed that the group with higher HIT approval rates (i.e., 99%--100%) showed a greater magnitude effect and were more likely to pass the quality control methods than those with lower rates. In addition, the majority of the quality control methods were able to preserve the magnitude effect at the expense of sample sizes. The best combinations of quality control methods that optimally maintained the effect were explored with a discussion of their implications. Although improving data quality is imperative, caution should be exercised as the quality control methods may potentially bias the sample in an undesirable manner (e.g., excluding individuals who show steep delay discounting). Keywords MTurk * Data quality * Online research * Delay discounting * Magnitude effect
Introduction The use of online crowdsourcing platforms to collect behavioral data has become popular in social science research in recent decades. These platforms provide evident scientific benefits, including efficiency and [...]