Crowdsourced science (not to be confused with citizen science, a subtype of crowdsourced science) refers to collaborative contributions of a large group of people to the different steps of the research process in science. In psychology, the nature and scope of the collaborations can vary in their application and in the benefits it offers.
What is crowdsourcing? A complement to the traditional way of doing science Crowdsourcing is a collaborative sourcing model in which a large and diverse number of people or organizations can contribute to a common goal or project.
First examples of crowdsourcing science can be found during the 19th century. A Yale University professor launched a call for open participation of citizens to maximize the number and diversity of observations he could get on the Leonids of 1833 meteor storm phenomenon.
Crowdsourced science has been set aside for a long time and has only recently gained popularity in science. Actually, it helps overcoming several limits of the classic model on which science has been conducted up till now. For example, in psychology, scientific research has often been limited by small sample sizes and a lack of diversity in studied populations. These limits can be tackled with a more collaborative approach of scientific research (i.e., crowdsourced science).
A two-dimensional concept Crowdsourcing initiatives can be described along two different continuous axes. The first dimension represents the degree of communication between project members, ranging from very few collaborators and little communication, to a large crowd of collaborators with a rich amount of communication. The second dimension corresponds to the degree of inclusiveness, varying from only selecting groups of people with a high expertise in the field of interest, to open to anyone—with or without expertise—interested in collaborating. This twofold distinction draws four types of crowdsourced science projects. However, these two dimensions lie on a continuum: A single project is neither entirely inclusive (vs. selective) nor entirely high (vs. low) in communication.
Why crowdsource psychological science? Limits of the traditional vertical model In psychology, the traditional way of doing research follows a vertical model. In other words, most of the time, research is carried out by small teams from a specific lab or university, and organized around one or two main researchers (often referred to as Principal Investigator, or PI). This small team contributes to all different stages of the research project: conception of the research question, design of the study, data collection, data analysis, discussion of the results, and publication of the manuscript.
However, vertical science has its own limitations that can impede the progress of science. Conducting science within small independent teams makes it difficult to conduct large-scale projects (with large samples, large databases and high statistical power), which may limit the scope of research. In traditional science, researchers have access to less resources, data, and methodologies. Small, independent team projects are also limited in the feedback they can get from other perspectives.
Nowadays, research teams do not necessarily have to make compromises anymore (e.g., small sample size, same stimuli, same contexts and no replication), because much of these limits could be tackled by a more crowdsourced research.
The replication crisis In the scientific method, replicability is a fundamental criterion to qualify a theory as scientific. The replication crisis (or credibility crisis) is a methodological crisis in science that researchers began to acknowledge around the 2010s. The controversy revolves around the lack of reproducibility of many scientific findings, including those in psychology (e.g., among 100 studies, less than 50% of the findings were replicated).
Some of its main underlying causes are referred to as "questionable research practices". These include p-hacking (i.e., exploiting researcher degrees of freedom until a significant result is obtained), HARKing (i.e., hypothesizing after results are known), publication bias (i.e., the tendency among scholarly journals to only publish significant results), statistical reporting errors, and low statistical power often due to small sample sizes.
Among these various issues, small sample sizes and the lack of diversity within samples can be addressed through crowdsourced science—increasing the generalizability of findings and therefore their replicability as well. Indeed, samples in psychology often rely on college students and Western, Educated, Industrialized, Rich and Democratic (WEIRD) populations.
For these reasons, the replication crisis has contributed to the rise of crowdsourced, large-scale projects, especially replication projects held at an international scale like the Many Labs project, and the Psychological Science Accelerator (PSA). These crowdsourced projects aim at solving some issues raised by the replication crisis, more specifically by assessing the replicability of studies and generalisation of the results to other populations and contexts.
Ambitions of the horizontal model In contrast to the vertical model of doing research, the horizontal model—an inherent principle of crowdsourced science—mainly relies on variations in terms of inclusiveness and communication. Its core principle is about the non-authority of one or two researchers in terms of resources, ownership and expertise. Following this principle, the different tasks within a project are distributed among many researchers. The whole project is then supported by a team, which ensures the coordination of the contributors.
A perfect horizontal model does not really exist because vertical and horizontal models are more conceptualized as extremes of a same continuum. This distributed model of science work is gaining popularity in the scientific community. In about 40 years and across different scientific disciplines, research teams have roughly doubled in size (from 2 to 4 people on average).
By encouraging larger crowds to contribute to research projects in psychology, the horizontal model aims at reducing some limits of the vertical one. It has three distinct ambitions:
Carry out wide-ranging works Encourage a democratized psychological science Establish robust findings
Large-scale research projects The first ambition of the horizontal model is to enable researchers to conduct more large-scale research projects (i.e., ambitious projects that cannot be conducted by small teams). By aggregating various skills and resources, it is possible to move from a model where research is defined by available means, to a model where research itself defines the necessary means to answer the research question.
Democratizing psychological science The second ambition of the horizontal model is to compensate for inequalities (e.g., in terms of recognition, status, and success in a researcher's career). Psychology and more generally social sciences show a strong bias of what is called the Matthew effect (i.e., academic advantages going to those who are already the most renowned).
Early career researchers from less renowned institutions, less economically developed countries or underrepresented demographic groups are generally less likely to have access to high-profile projects. Crowdsourcing enables such researchers to contribute to impactful projects and gain recognition for their work.
Robust findings The third ambition of the horizontal model is to improve the robustness, generalizability, and reliability of findings in order to increase the credibility of psychological science. Horizontal collaboration between research teams facilitates the replication of studies and makes it easier to detect biased effect sizes, p-hacking, and publication bias—different problems raised by the replication crisis (see also #The replication crisis).
Crowdsourcing in practice Contributions at different stages of research This section aims at detailing how crowdsourcing can contribute to the different stages of the research process, from the generation of research ideas to the publication of the outcomes.
Ideation Ideation is the first step of any research project. In psychology, it refers to the process of defining the general idea behind a project—purpose, research question, and hypotheses. This step can be done in collaboration between several researchers to scan a broader spectrum of ideas and select those of broadest interest and impact. Faced with a research question, the different collaborators can bring their expertise in the construction of hypotheses.
Crowds can also help to generate new ideas to solve complex-problems, such as illustrated by the Polymath project.
Assembling resources When assembling resources, crowdsourcing can be useful, especially for labs with less resources at their disposal. Online platforms such as Science Exchange and Studyswap allow researchers to establish new communication lines and share resources between labs. Matching resources from labs across the globe minimizes waste and facilitates research teams' ability to meet their goals.
Sharing tasks across labs can improve the efficiency of a research project, especially highly time-consuming ones. In biology, for example, studying the entire genome takes a lot of time. By distributing its investigation and combining resources across multiple labs, it is possible to accelerate the research process.
Study design There can be many ways to design a study. Research teams across the world neither have the same theoretical background, nor are they all equipped with the same materials.
Crowdsourcing can be useful in the case of conceptual replications (i.e., testing a same research question through different operationalizations). When testing a same research question, variations in study designs can lead to strong variations in effect size estimations. Diversifying the methods used to test the same hypothesis across different populations—through collaborative projects—enables better estimations of the true consistency of a scientific claim.
Data collection In psychology, data collection often relies on samples drawn from Western, Educated, Industrialized, Rich and Democratic (WEIRD) populations, which impedes the overall generalizability of findings. Crowdsourcing the data collection process by relying on multi-lab data collection and online crowdsourcing platforms (e.g., Amazon Mechanical Turk, Prolific) makes it easier to reach a wider audience of participants from different cultural backgrounds and non-WEIRD populations. When the research question makes it possible to rely on internet samples, it is also an easy way to recruit larger samples of participants with minimal financial input and within short amounts of time.
Most of the time, members of the general public are recruited to undergo studies as research participants but they can also be recruited to collect data and observations.
Data analysis In research, data analysis refers to the process of cleaning, transforming, and modeling data using statistical tools, often with the purpose of answering a research question. Within a research project, this is typically done by a single analyst (or team) and results in a single analysis of a dataset.