Mark Rubin

Preregistration entails researchers registering their planned research
hypotheses, methods, and analyses in a time-stamped document before they
undertake their data collection and analyses. This document is then made
available with the published research report in order to allow readers
to identify discrepancies between what the researchers originally
planned to do and what they actually ended up doing. In a recent article
([Rubin, 2020](https://doi.org/10.20982/tqmp.16.4.p376)), I question
whether this *historical transparency* facilitates judgments of
credibility over and above what I call the *contemporary transparency*
that is provided by (a) clear rationales for current hypotheses and
analytical approaches, (b) public access to research data, materials,
and code, and (c) demonstrations of the robustness of research
conclusions to alternative interpretations and analytical approaches.

<figure>
<img
src="https://cdn-images-1.medium.com/max/402/1*YYny0udOTH_iiN4dieeCmA.png" />
</figure>

My article covers issues such as
[HARKing](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-costs-of-harking),
multiple testing, *p*-hacking, [forking
paths](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/the-garden-of-forking-paths),
[exploratory
analyses](https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/p-values-in-exploratory-analyses),
optional stopping, researchers' biases, selective reporting, test
severity, publication bias, and replication rates. I argue for a nuanced
approach to these issues. In particular, I argue that only *some* of
these issues are problematic, and only under *some* conditions. I also
argue that, when they are problematic, these issues can be identified
via contemporary transparency per se.

<figure>
<img
src="https://cdn-images-1.medium.com/max/712/1*W15_55y4m_aPZ1E90yp5AA.png" />
</figure>

I conclude that preregistration's historical transparency does not
facilitate judgments about the credibility of research findings when
researchers provide contemporary transparency. Of course, in many cases,
researchers do not provide a sufficient degree of contemporary
transparency (e.g., no open research data or materials), and in these
cases preregistration's historical transparency may provide some useful
information.

However, I argue that historical transparency is a relatively narrow,
researcher-centric form of transparency because it focuses attention on
the predictions made by specific researchers at a specific point in
time. In contrast, contemporary transparency allows research data to be
considered from multiple, unplanned, theoretical and analytical
perspectives while maintaining a high degree of research credibility.
Hence, I suggest that the open science movement should push more towards
contemporary transparency and less towards historical transparency.

<figure>
<img
src="https://cdn-images-1.medium.com/max/1024/1*Dk63b8_EVkeBDgMbu9fjQg.png" />
</figure>

For further information, please see:

Rubin, M. (2020). Does preregistration improve the credibility of
research findings? *The Quantitative Methods in Psychology, 16*(4),
376--390. <https://doi.org/10.20982/tqmp.16.4.p376>

I was invited to present my paper at the opening cermony of the Erasmus
Research Institute of Management Research Transparency Campaign at
Erasmus University Rotterdam on 1st March 2022. A recording of my
presentation is below:

<https://medium.com/media/426ebd8a39df3d5e3d87e9c18b29bb7b/href>

For more of my work in this area, please see:
<https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis>

![](https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4e872ce8a3c){width="1"
height="1"}


Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report in order to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing.

Does Preregistration Improve the Interpretablity and Credibility of Research Findings?

Björn Brembs

In the
[last](https://www.theguardian.com/science/audio/2015/jun/12/scientific-retractions-fraud-explored)
"Science Weekly" podcast from the Guardian, the topic was retractions. 
At about 20:29 into the episode, [Hannah
Devlin](https://www.theguardian.com/profile/hannah-devlin) asked,
whether the reason 'top' journals retract more articles may be because
of increased scrutiny there.

The underlying assumption is very reasonable, as many more eyes see each
paper in such journals and the motivation to shoot down such
high-profile papers might also be higher. However, the question has
actually been addressed in the scientific literature and the data don't
seem to support this assumption. For one, [this
figure](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3479492/figure/fig03/?report=objectonly){target="_blank"}
shows that there are a lot of retractions from lower ranking journals,
but the journals who retract a lot are few and far between. In fact,
there are many more retractions in low-ranking journals than in
high-ranking ones. Of the high-ranking journals, a much larger
proportion also retracts many papers. However, this analysis only shows
that there are many more retractions in lower journals than in higher
journals on an absolute level. Hence, these data are not conclusive, but
suggestive that scrutiny is not really all that much higher for the
'top' journals than anywhere else.

Another reason why scrutiny might be assumed to be higher in 'top'
journals is that readership is higher, leading to more potential for
error detection. However, the same reasoning can be applied to
citations, and not only retractions. Moreover, citing a 'top' paper is
not only easier than forcing a retraction, it also benefits your own
research by elevating the perceived importance of your field. Thus, if
readership had any such influence, one would expect journal rank to
correlate better with citations than with retractions. The opposite is
the case: The coefficient of determination for citations with journal
rank currently lies around 0.2, while that coefficient comes to lie at
just under 0.8 for retractions and journal rank (Fig. 3 and Fig. 1D,
respectively,
[here](https://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full)).
So while there may be a small effect of scrutiny/motivation, the
evidence seems to suggest that it is a relatively minor effect, if there
is one at all.

Conversely, there is quite solid evidence that the methodology in 'top'
journals is not any better than in other journals, when analyzing
non-retracted articles. In fact, there are studies showing that the
methodology is actually *worse* in 'top' journals, while we have not
found a single study suggesting the methodology gets better with journal
rank. Our
[article](https://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full)
reviews these studies. Importantly, these studies all concern
non-retracted papers, i.e., the other 99.95% of the literature.

In conclusion, the evidence suggests scrutiny is likely a negligible
factor in the correlation of journal rank and retractions, while
increased incidence of fraud and lower methodological standards can be
shown.

I know [Ivan
Oransky](https://retractionwatch.com/meet-the-retraction-watch-staff/about/),
who was a guest on the show, is aware of these data, so it may have been
a bit unfortunate that Phillip Campbell (editor-in-chief at Nature
Magazine) got to answer this question before Ivan had a chance to lay
these data out. In fact, Nature is also aware of these data and has
twice refused publishing them. The first time when we submitted our
manuscript, with the statement, that Nature had already published
articles that stated that [Nature publishes the worst
science](https://bjoern.brembs.net/2013/06/everybody-already-knows-journal-rank-is-bunk).
The second time was when Cori Lok interviewed Jon Tennant and he told
her about the data, but Cori [failed to include this
part](https://bjoern.brembs.net/2014/09/how-nature-magazine-consistently-prefers-anecdote-over-data)
of the interview. There is thus a record of Nature, very understandably,
avoiding to admit their failure to select for solid science. Phillip
Campbell's answer to the question in the podcast may have been at least
the third time.

While Phillip Campbell did admit they don't do enough fraud-detection
(it is too rare), the issue of reliability in science goes far beyond
fraud, so successfully derailing the question towards this direction
served his company quite well. Clearly, he's a clever guy and did not
come unprepared.

Finally, one may ask: why do the 'top' journals publish unreliable
science?

Probably the most important factor is that they attract "too good to be
true" results, but only apply "peer-review light": rejection rates drop
dramatically from 92% to a mere 60% once your manuscript makes it past
the editors, that's a 5-fold increase in your publication chances (Noah
Gray and Henry Gee, pers. comm.). Why is that so? First, the reviewers
know the editor wants to publish this paper. Second, they have an
automatic conflict of interest, as a Nature paper in their field
increases the visibility of their field, they may even be cited in the
paper -- or plan to cite it in their upcoming grant application.

On average, this entire model is just a recipe for disaster and more
policing won't fix it. By using it, we have been setting us up for the
exponential rise in retractions to be seen in [Fig. 1a of our
paper](https://www.frontiersin.org/files/Articles/45406/fnhum-07-00291-r2/image_m/fnhum-07-00291-g001.jpg).

So, in the probably not too unlikely case that the topic of unreliable
science should come up again, anyone can now cite the actual,
peer-reviewed data we have at hand, such that editors-in-chief may have
a harder time derailing the discussion and obfuscating the issues in the
future.

**tl;dr:** The data suggest a combination of three factors leading to
more retractions in 'top' journals: 1. Worse methodological quality; 2.
Higher incidence of fraud 3. Peer-review light. One would intuitively
expect increased readership/scrutiny to play some role, but there is
currently no evidence for it and some circumstantial evidence against
it.


In the last “Science Weekly” podcast from the Guardian, the topic was retractions.  At about 20:29 into the episode, Hannah Devlin asked, whether the reason ‘top’ journals retract more articles may be because of increased scrutiny there. The underlying assumption is very reasonable, as many more eyes see each paper in such journals and the motivation to shoot down such high-profile papers might also be higher.

Postagens de Rogue Scholar

Does Preregistration Improve the Interpretablity and Credibility of Research Findings?

Are more retractions due to more scrutiny?