A brief intro for research students.

Good science is cumulative

Scientists seek to create new knowledge, often by conducting an experiment or other research study.

But science is much more than doing studies and analyzing the data. Critical to the scientific enterprise is communication of what was done, and what was found.

Isaac Newton, who formulated the laws of motion and gravity, wrote that “If I have seen further it is by standing on the shoulders of giants.” Newton knew that science is cumulative – we build on the findings of previous researchers.

Robert Merton, a sociologist of science, described values or norms that are endorsed by many scientists. One of these that is critical to ensuring that science is cumulative is the norm of communalism.  Communalism refers to the notion that scientific methodologies and results are not the property of individuals, but rather should be shared with the world.

Sharing allows others to know the details of a previous study, which is important for:

1. Understanding the study’s results

2. Building on its methodology

3. Confirming its findings

This last purpose is, arguably, essential. But across diverse sciences, ensuring that confirmation can be done, as well as actually doing it, has been neglected. This is the issue of reproduciblity.

Reproducibility

Another scientific norm important to achieving reproducibility was dubbed organized skepticism by Merton. The critical community provided by other researchers is thought to be key to the success of science. The Royal Society of London for Improving Natural Knowledge, more commonly known as simply the Royal Society, was founded in 1660 and established many of the practices we today associate with science worldwide. The Latin motto of the Royal Society, “Nullius in verba”, is often translated as “Take nobody’s word for it”. 

Anyone can make a mistake, and most or all of us have biases, so scientific claims should be verifiable. The historian of science David Wooton has written that “What marks out modern science is not the conduct of experiments”, but rather “the formation of a critical community capable of assessing discoveries and replicating results.”

Types of reproducibility

Assessing discoveries and replicating results can involve two distinct types of activities. One can include examining the records associated with a study to check for errors. The second involves attempting to re-do the study and see whether similar data results that support the original claim.

The first type of activity is often referred to today with the phrase computational reproducibility. The word “computational” refers to taking the raw observations or data recorded for a study and re-doing the analysis that most directly supports the claims made.

The second activity is often referred to as replication. If a study is redone by collecting new data, does this replication study yield similar results to the original study? If very different results are obtained, this may call into question the claim of the original study, or indicate that the original study is not one that can be easily built upon.

Sometimes the word empirical is put in front of reproducibility or replication to make it clear that new data are being collected, rather than referring to computational reproducibility.

The replication crisis

The importance of reproducibility, in principle, has been recognized as critical throughout the history of science. In practice, however, many sciences have failed to adequately incentivize replication. This is one reason (later we will describe others) for the replication crisis.

The replication crisis refers to the discovery, and subsequent reckoning with, the poor success rates of efforts to computationally reproduce or empirically replicate previous studies.

The credibility revolution

The credibility revolution refers to the efforts by individual researchers, societies, scientific journals, and research funders to improve reproducibility. This has led to

1. New best practices for doing individual studies

2. Changes in how researchers and their funding applications are evaluated

3. Greater understanding of how to evaluate the credibility of the claims of individual studies

The word credibility refers to how believable a theory or claim is. This reflects both how plausible it is before one hears of any relevant evidence, plus the evidence for the theory. Thus if a claim is highly credible, the probability that it is true is high.  The phrase credibility revolution helps convey that reforms related to reproducibility have boosted the credibility of many scientific theories and claims.