The seriousness check is a technique that can be used in online research (also called Internet-based research, Web-based research, Web-based experiments) to improve data quality. Nowadays, many scientific studies with human participants are conducted online and are accessible to a large diversity of participants. Nonetheless, many people just want to look at the different pages of the questionnaire, instead of giving carefully chosen answers to the questions.Reips, U.-D. (2009). Internet experiments: Methods, guidelines, metadata. Human Vision and Electronic Imaging XIV, Proc. SPIE, 7240, 724008. https://doi.org/10.1117/12.823416 The seriousness check addresses this problem: In this approach the respondents are asked about the seriousness of their participation or for a probability estimate that they will complete the entire study or experiment.Musch, J., & Klauer, K. C. (2002). Psychological experimenting on the World Wide Web: Investigating content effects in syllogistic reasoning. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.), Online social sciences (pp. 181–212). Hogrefe & Huber Publishers.Reips, U.-D. (2007). The methodology of Internet-based experiments. In A. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford Handbook of Internet Psychology (pp. 373-390). Oxford University Press.Reips, U.-D. (2002). Standards for internet-based experimenting. Experimental Psychology, 49(4), 243–256. https://doi.org/10.1026/1618-3169.49.4.243 Thus, by using the seriousness check irrelevant data entries can be easily identified and be excluded from the data analysis.
Method
Seriousness checks can be implemented both before and after Aust, F., Diedenhofen, B., Ullrich, S., & Musch, J. (2013). Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45(2), 527–535. https://doi.org/10.3758/s13428-012-0265-2 participation in the study. However, it has been shown that the seriousness check is a good predictor of dropout rates when implemented in the first page of the experiment.
There are Web-based tools, e.g. WEXTOR, that implement the seriousness check by default.
Impact
Several studies have shown that performing a seriousness check at the start of a study best predicts motivation and dropout probability.Reips, U.-D. (2008). How Internet-mediated research changes science. In A. Barak (Ed.), Psychological aspects of cyberspace: Theory, research, applications (p. 268–294). Cambridge University Press. https://doi.org/10.1017/CBO9780511813740.013 It was observed that of those answering “I would like to look at the pages only" around 75% will drop, while of those answering “I would like to seriously participate now" only ca. 10-15% will drop during the study. Overall, about 30-50% of visitors will fail the seriousness check, i.e. answer “I would like to look at the pages only". Moreover, it was found that emphasizing seriousness increased information seeking in participants and the time they spent on the study. Following up on this, it was shown that motivation and self-reported seriousness significantly predict several data quality indicators
References