Collecting arguments on the stigma surrounding retraction

Reese Richardson

*Header image by Sydney Richardson.*

On June 5, head of Research Integrity and Resolutions at Springer Nature
[Tim Kersjes](https://twitter.com/TimKersjes) presented "Requiem for the
Retraction: why proposals for 'honest retractions' fail in their
objective to promote and encourage the self-correction of science" at
the 2024 World Conference on Research Integrity (WCRI).

Included was the following slide, later shared by [Maria
Zalm](https://x.com/MariaZalm/status/1798304619288047661), senior editor
and team manager of Publication Ethics at PLOS:

<figure class="wp-block-image size-large">
<img
src="https://reeserichardson.blog/wp-content/uploads/2024/06/cropped.jpg?w=1024"
class="wp-image-269" data-attachment-id="269"
data-permalink="https://reeserichardson.blog/2024/06/18/how-shameful-should-retraction-be/cropped/"
data-orig-file="https://reeserichardson.blog/wp-content/uploads/2024/06/cropped.jpg"
data-orig-size="1955,1100" data-comments-opened="1" data-image-meta="{"
aperture":"0","credit":"","camera":"","caption":"","created_timestamp":"0","copyright":"","focal_length":"0","iso":"0","shutter_speed":"0","title":"","orientation":"0"}"=""
data-image-title="cropped" data-image-description=""
data-image-caption=""
data-medium-file="https://reeserichardson.blog/wp-content/uploads/2024/06/cropped.jpg?w=300"
data-large-file="https://reeserichardson.blog/wp-content/uploads/2024/06/cropped.jpg?w=1024"
tabindex="0" role="button" loading="lazy" width="1024" height="576" />
<figcaption>Slide from Tim Kersjes’ talk at WCRI
2024.</figcaption>
</figure>

> More retraction positivity, less retraction shaming
>
> More and more and more and more
>
> **You want science to be more self-correcting? Stop retraction
> shaming!**
>
> Retractions are normal part of academic publishing and that is **[a
> good thing]{.underline}**
>
> Emphasize that retractions are a signal that editors and publishers
> **[are taking their responsibility]{.underline}**
>
> **[Be careful to jump to conclusions]{.underline}** when you see that
> an article is retracted
>
> Encourage editors to consider that retractions are **[a great
> way]{.underline}** to correct the published record
>
> More importantly: emphasize to authors that they are **[not being
> punished]{.underline}**

Reactions followed. Pseudonymous sleuth
[Morty](https://twitter.com/mortenoxe/status/1798771972555366889) said:

> If this is the final conclusion from \#WCRI2024, so help us God. Most
> of the retractions is due to research misconduct. We need more shaming
> of scientists who commit research misconduct!

Reproducibility researcher and science writer [James
Heathers](https://twitter.com/jamesheathers/status/1801764730031284493)
said:

> I was just thinking about 'stop retraction-shaming' today. While
> reading a 15 month old email thread pleading with an editor to retract
> clearly falsified medical papers.

Sleuth and founder of the 5GH Foundation [WU
Guangheng](https://twitter.com/Dr_5GH/status/1798410890125213868) said:

> We do NOT agree. Honest errors have a very small number, but hundreds
> of thousands of misconduct cases are waiting publishers to handle.
> Both PLoS and Springer Nature should take more effort to "clean up"
> their history, and \[quickly respond\] to new cases.

Maria Zalm
[responded](https://x.com/MariaZalm/status/1798830107425157549):

> To clarify, \[Tim Kersjes\] and I are not suggesting that fraudulent
> research should not be retracted, but rather that unreliable articles
> should be retracted REGARDLESS of misconduct findings.
>
> ...
>
> \[If\] we are truly serious about correcting the scientific
> literature, we need to encourage people to retract their own
> erroneous/unrepeatable results, and this won't happen as long as there
> is such a huge \[stigma\] around all retractions.

Other have written at length about the stigma surrounding retractions:

1. [Ending the retraction stigma: Encouraging the reporting of errors
 in the biomedical record](https://doi.org/10.1177/1747016118802970)
2. [Breaking the stigma of
 retraction](https://doi.org/10.1038/s41562-021-01266-7)
3. [Retraction Stigma and its Communication via Retraction
 Notices](https://doi.org/10.1007/s11024-022-09465-w)

In my own research and in discussions with others, I have encountered
compelling arguments both for why retraction should be more stigmatized
and why retraction should be less stigmatized. I have yet to form a
coherent opinion myself. Here, I wanted to collect and discuss these
arguments to aid my own understanding and invite arguments from others.

We should first frame this debate in terms of a common value: the
essentiality of retractions. Most observers will agree that retractions
are an indispensable part of the scientific process and that it is a
good thing when an unreliable paper is retracted. Most observers will
agree that even though retraction is becoming [increasingly
frequent](https://doi.org/10.1038/d41586-023-03974-8), not enough papers
are retracted and retractions often take far too long. One might argue
that retraction is not an essential part of science publishing (see
["Science is a strong-link
problem"](https://www.experimental-history.com/p/science-is-a-strong-link-problem)
by psychologist Adam Mastroianni), but we need to stick to one framework
by which we can judge our arguments. Thus, we should prefer a position
that improves the efficiency and integrity of retraction as a research
publishing practice.

### Position A: Retractions should be more stigmatized {#position-a-retractions-should-be-more-stigmatized .wp-block-heading}

###### Contention A1: Most retractions are due to misconduct and misconduct should be shameful {#contention-a1-most-retractions-are-due-to-misconduct-and-misconduct-should-be-shameful .wp-block-heading}

Observers agree that most retractions are the result of research
misconduct on the part of one or more authors ([Fang et
al.](https://doi.org/10.1073/pnas.1212247109), [Freijedo-Farinas et
al.](https://doi.org/10.1007/s11192-024-04992-7), [Grieneisen and
Zhang](https://doi.org/10.1371/journal.pone.0044118)). Many retractions
are now the result of particularly egregious forms of misconduct like
paper milling or other systematic research fraud. Research misconduct
should not be tolerated. As such, instances of misconduct should be met
with scorn for the sake of punitive and preventive justice.

There is some evidence that retraction shaming, as an attitudinal
implementation of preventive justice, would work as a deterrent against
research misconduct. [Chen et
al.](https://doi.org/10.1186/s12909-024-05277-6) report that lower
perceived consequences were positively associated with self-reporting of
research misconduct in a large sample of Chinese physicians. [Martinson
et al.](https://doi.org/10.1525/jer.2006.1.1.51) suggest that
scientists' perceptions of organizational injustice may also contribute
to misconduct.

###### Contention A2: Shameless repeat offenders will continue their careers {#contention-a2-shameless-repeat-offenders-will-continue-their-careers .wp-block-heading}

Researchers that have published retracted work can and do lose their
jobs and access to funding, even without official findings of misconduct
from their employers. However, if employers are not diligent in their
background checks or simply do not care, the most ignominious offenders
might be able to keep their jobs or move onto new ones and continue
recklessly polluting the scientific record. I surmise that researchers
that are comfortable violating ethical norms of scientific conduct would
also be more likely to violate professional and personal ethical norms.

Shame should be used to prevent these individuals from continuing to
climb the ranks for protection of the scientific record and possibly the
protection of violators' colleagues, students and patients. An official
finding of misconduct will most likely not appear in a Google search of
a quack doctor's name. A tweet will.

###### Contention A3: Shame is all we have {#contention-a3-shame-is-all-we-have .wp-block-heading}

Retractions happen very infrequently; there were about 10,000 published
retractions to about 10,000,000 published peer-reviewed articles in 2023
(a 1:1,000 ratio). Many retracted articles are not prominently labelled
as retracted by the publisher [despite COPE
guidelines](https://doi.org/10.24318/cope.2019.1.4). Lots of retracted
articles will continue to be cited as if they were not retracted (see
[Hsiao and Schneider](https://doi.org/10.1162/qss_a_00155)). Moreover,
retracted articles are still counted towards bibliometrics like h-index
and citation count in most databases. Beyond retraction, individual
sanctions are exceedingly infrequent. For instance, the [US Department
of Health and Human Services Office of Research Integrity
sanctions](https://ori.hhs.gov/content/case_summary) on the order of 10
individuals every year, often fewer. University investigations often
close without official misconduct findings even as retractions mount.

If these formal measures are woefully insufficient for addressing
research misconduct, we should practice shame as a low-cost and scalable
alternative.

### Position B: Retractions should be less stigmatized {#position-b-retractions-should-be-less-stigmatized .wp-block-heading}

###### Contention B1: Retraction stigma could prevent self-reported retractions {#contention-b1-retraction-stigma-could-prevent-self-reported-retractions .wp-block-heading}

In most cases, authors will be those that most completely understand the
content of their scientific work. Thus, authors are among the
best-prepared to spot errors in their work and move to correct or
retract. A heavy stigma and career penalty associated with retractions,
real or perceived, could discourage authors from retracting their own
work. This might explain why so few retractions are associated with
honest error, as mentioned in Contention A1.

In 2020, Chemistry Nobel laureate Frances Arnold retracted a [2019
*Science* article](https://doi.org/10.1126/science.aaw9068) she
authored,
[noting](https://retractionwatch.com/2020/01/02/nobel-winner-retracts-paper-from-science/):

> For my first work-related tweet of 2020, I am totally bummed to
> announce that we have retracted last year's paper on enzymatic
> synthesis of beta-lactams. The work has not been reproducible.
>
> It is painful to admit, but important to do so. I apologize to all. I
> was a bit busy when this was submitted, and did not do my job well.

Many commentators (rightfully, in my opinion) celebrated Arnold's
transparency and proactiveness. Early-career researchers without a Nobel
prize or a well-earned reputation might be more reluctant to follow this
path exactly because of the perceived personal cost.

###### Contention B2: Retraction stigma impedes the retraction process {#contention-b2-retraction-stigma-impedes-the-retraction-process .wp-block-heading}

Authors will often mount vicious defenses against criticisms and
investigations of their work that could end in retraction. The
perception that a retraction will bring public and personal shame will
undoubtedly fuel this behavior. The threat of shame can prompt authors
to launch lawsuits, spout vitriol or not cooperate with investigations
by journals and employers, tying up proceedings and delaying retractions
for years. James Heathers is not alone in his 15-month quest to get a
paper retracted---experienced sleuths will have endured innumerable
prolonged matches of back-and-forth with editors and authors. Fear of
getting tangled up in these matches will also discourage observers from
reporting errors in the first place.

Editors will be similarly disincentivized against initiating
investigations and executing retractions by the perception of shame. No
editor would want the name of their journal besmirched by calling
attention to a problematic paper with a retraction. Similarly, editors
will want to maintain professional relationships with prestigious
researchers. This is hard to do if the editor is perceived as an agent
of a humiliating witch hunt.

###### Contention B3: Shame can be misdirected {#contention-b3-shame-can-be-misdirected .wp-block-heading}

Retraction notices rarely assign blame for mistakes or misconduct. Many
retraction notices are cartoonishly opaque (e.g. ["\[the\] article...has
been voluntarily retracted by the authors due to a 'non-scientific
reason'"](https://doi.org/10.1096/fj.201700932RRRET)). As a result, it
is very difficult for readers to properly evaluate where shame should be
directed or if it should be employed at all. Obscurant practices by
institutional committees charged with investigating misconduct compound
this difficulty. In this fog, there is always a strong possibility that
the innocent will be targeted for shaming and the guilty will escape
ire.

Junior researchers are already the most likely to shoulder the brunt of
retraction's consequences. [Jin et
al.](https://doi.org/10.1162/rest_a_00780) found that citation losses to
prior works by an author following a retraction of their work are far
less severe for eminent authors. The citation penalty on prior work
(previously described [by the same
team](https://doi.org/10.1038/srep03146)) also implies that other,
unimpeachable work by a group of authors will earn fewer citations
following the retraction of one of their otherwise unrelated articles.
In essence, shame might cause us to disregard valuable work.

How much collateral damage should the scientific community tolerate in
the name of punishing misdeeds?

###### Contention B4: There is not enough shame to go around {#contention-b4-there-is-not-enough-shame-to-go-around .wp-block-heading}

Attention spans are limited. Cultural memory can be short. Even the most
tireless cranks must eventually stop ringing the alarm. There are
undoubtedly tens of thousands of scientific fraudsters out there who
deserve to lose face and never will simply because shame is
rate-limited.

In our cultural context, this also means that the most shame will be
reserved for the most scandalous cases. Hence, headlines will abound for
one [AI-generated rat
penis](https://youtu.be/Bj8IAoTnyNw?si=dViOmf0ZYMm-eGuV) but a case of
[more than 50 medical studies with faked
authorship](https://retractionwatch.com/2024/01/26/journal-retracts-more-than-50-studies-from-saudi-arabia-for-faked-authorship/)
will get comparatively little attention. If we aim the weapon of shame
selectively and illogically, we should hesitate to wield it.

### Position C: We are asking the wrong question {#position-c-we-are-asking-the-wrong-question .wp-block-heading}

The more practically-minded among us will note that we are debating
attitudinal solutions to what is clearly a structural problem.

Not only do these solutions have duller teeth than sweeping systemic
changes, but no one has the authority to force a cultural change in how
something is stigmatized. As retractions become more frequent, it is
also difficult to predict how that stigma might change on its own. We
should direct our effort towards reshaping scientific institutions for
organizational justice instead of litigating how scientists should feel
about science integrity measures.

### What are your thoughts? {#what-are-your-thoughts .wp-block-heading}

Feel free to share your thoughts and arguments for or against retraction
shaming in the comments on this post. As retractions grow exponentially,
discussions like these will shape the future of scientific publishing
and may alter the character of 'self-correction' as a scientific maxim.

Header image by Sydney Richardson.

On June 5, head of Research Integrity and Resolutions at Springer Nature Tim Kersjes presented “Requiem for the Retraction: why proposals for ‘honest retractions’ fail in their objective to promote and encourage the self-correction of science” at the 2024 World Conference on Research Integrity (WCRI). Included was the following slide, later shared by Maria Zalm, senior editor and team manager of

How shameful should retraction be?

Björn Brembs

[***tl;dr:** Data from thousands of non-retracted articles indicate that
experiments published in higher-ranking journals are less reliable than
those reported in 'lesser' journals.*]{style="font-size: 10pt;"}

[Vox](https://www.vox.com) health reporter [Julia
Belluz](https://www.vox.com/authors/julia-belluz) has recently covered
the reliability of
[peer-review](https://www.vox.com/2015/12/7/9865086/peer-review-science-problems).
In her follow-up piece, she asked "[Do prestigious science journals
attract bad
science?](https://www.vox.com/2016/1/11/10749636/science-journals-fraud-retractions)".
However, she only covered the data on retractions, not the much less
confounded data on the remaining, non-retracted literature. It is indeed
interesting how everyone seems to be attracted to the retraction data
like a moth to the flame. Perhaps it's because retractions constitute a
form of 'capital punishment', they seem to reek of misconduct or
outright fraud, which is probably why everybody becomes so attracted --
and not just journalists, scientists as well, I must say. In an email,
she explained that for a lay audience, retractions are of course much
easier to grasp than complicated, often statistical concepts and data.

However, retractions suffer from two major flaws which make them rather
useless as evidence base for any policy:

I. They only concern about .05% of the literature (perhaps an
infinitesimal fraction more for the 'top' journals
![🙂](https://s.w.org/images/core/emoji/14.0.0/72x72/1f642.png){.wp-smiley
style="height: 1em; max-height: 1em;"}  
II. This already unrepresentative, small sample is further confounded by
error-detection variables that are hard to trace.

Personally, I [tentatively
interpret](https://bjoern.brembs.net/2015/06/are-more-retractions-due-to-more-scrutiny)
what scant data we have on retractions as suggestive that increased
scrutiny may only play a minor role in a combination of several factors
leading to more retractions in higher ranking journals, but I may be
wrong. Indeed, we emphasize in several places in our
[article](https://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full)
on precisely this topic that retractions are rare and hence one
shouldn't place so much emphasis on them, e.g.:  
"These data, however, cover only the small fraction of publications that
have been retracted. More important is the large body of the literature
that is not retracted and thus actively being used by the scientific
community."  
Given the attraction of such highly confounded data, perhaps we should
not have mentioned retraction data at all. Hindsight being 20/20 and all
that...

Anyway, because of these considerations, the majority of our
[article](https://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full)
is actually about the data concerning the non-retracted literature
(i.e., the other 99.95%). In contrast to retractions, these data do not
suffer from any of the above two limitations: we have millions and
millions of papers to analyze and since all of them are still public,
there is no systemic problem of error-detection confounds.

For instance, we review articles that suggest that (links to articles in
our
[paper](https://journal.frontiersin.org/article/10.3389/fnhum.2013.00291/full)):

1\. Criteria for evidence-based medicine are no more likely to be met in
higher vs. lower ranking journals:  
Obremskey et al., 2005; Lau and Samman, 2007; Bain and Myles, 2005;
Tressoldi et al., 2013

2\. There is no correlation between statistical power and journal rank
in neuroscience studies:  
[Figure
2](https://www.frontiersin.org/files/Articles/45406/fnhum-07-00291-r2/image_m/fnhum-07-00291-g002.jpg):

[![Fig-2](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/Fig-2-1024x709.png){.aligncenter
.wp-image-1083 .size-large loading="lazy" decoding="async" width="520"
height="360"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/Fig-2-1024x709.png 1024w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/Fig-2-300x208.png 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/Fig-2-768x532.png 768w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/Fig-2-50x35.png 50w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/Fig-2.png 1595w"
sizes="(max-width: 520px) 100vw, 520px"}](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/Fig-2.png)

3\. Higher ranking journals tend to publish overestimates of true effect
sizes from experiments where the sample sizes are too low in
gene-association studies:  
[Figure
1C](https://www.frontiersin.org/files/Articles/45406/fnhum-07-00291-r2/image_m/fnhum-07-00291-g001.jpg):

[![effectsizeoverestimation](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/effectsizeoverestimation-1024x593.png){.aligncenter
.wp-image-1084 .size-large loading="lazy" decoding="async" width="520"
height="301"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/effectsizeoverestimation-1024x593.png 1024w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/effectsizeoverestimation-300x174.png 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/effectsizeoverestimation-768x445.png 768w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/effectsizeoverestimation-50x29.png 50w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/effectsizeoverestimation.png 1160w"
sizes="(max-width: 520px) 100vw, 520px"}](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/effectsizeoverestimation.png)

4\. Three studies analyzing replicability in biomedical research and
found it to be extremely low, not even top journals stand out:  
Scott et al., 2008; Prinz et al., 2011; Begley and Ellis, 2012

5\. Where quality can actually be quantified, such as in computer models
of crystallography work, 'top' journals come out significantly worse
than other journals:  
esp. Fig. 3 in Brown and Ramaswamy, 2007

![chrystallography](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/chrystallography.jpg){.aligncenter
.size-full .wp-image-1085 loading="lazy" decoding="async" width="599"
height="542"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/chrystallography.jpg 599w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/chrystallography-300x271.jpg 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/chrystallography-50x45.jpg 50w"
sizes="(max-width: 599px) 100vw, 599px"}  
After our review was published, a study came out which showed that

6\. *In vivo* animal experimentation studies are less randomized in
higher ranking journals and the outcomes are not scored more often in
blind in higher-ranking journals either:

![reporting](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/reporting.png){.aligncenter
.size-full .wp-image-1087 loading="lazy" decoding="async" width="872"
height="432"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/reporting.png 872w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/reporting-300x149.png 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/reporting-768x380.png 768w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/reporting-50x25.png 50w"
sizes="(max-width: 872px) 100vw, 872px"}

Hence, in these six (nine including the update below) areas,
unconfounded data covering orders of magnitude more material than the
confounded retraction data reveal only two out of three possible general
outcomes:

a\) Non-retracted experiments reported in high-ranking journals are no
more methodologically sound than those published in other journals.  
b) Non-retracted experiments reported in high-ranking journals are less
methodologically sound than those published in other journals

Not a single study we know of (there may be some we missed! Let me
know.) shows the third option of higher-ranking journals publishing the
**most** sound experiments. It is this third option that at least one
analysis should have found somewhere if there was anything to journal
rank with regard to reliability.

Hence, even if you completely ignore the highly scattered and confounded
retraction data, experiments published in higher ranking journals are
still less reliable than those published in lower ranking journals --
and error-detection or scrutiny has nothing to do with it.  
In that view, one may interpret the observation of more retractions in
higher ranking journals as merely a logical consequence of the worse
methodology there, nothing more. This effect may then, in turn, be
somewhat exaggerated because of higher scrutiny, but we don't have any
data on that.

All of this data is peer-reviewed and several expert peers
[attested](https://bjoern.brembs.net/2013/06/everybody-already-knows-journal-rank-is-bunk)
that none of the data in our review is in dispute. It will be
interesting to see if Ms. Belluz will remain interested enough to try
and condense such much more sophisticated evidence into a form for a lay
audience.
![🙂](https://s.w.org/images/core/emoji/14.0.0/72x72/1f642.png){.wp-smiley
style="height: 1em; max-height: 1em;"}

**UPDATE (9/9/2016)**: Since the publication of this post, two
additional studies have appeared that further corroborate the impression
that the highest ranking journals publish the least reliable science: In
the field of genetics, it appears that errors in gene names (and
accession numbers) introduced by the usage of Excel spreadsheets are
[more
common](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-016-1044-7)
in higher ranking journals:

[![excel_errors](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/excel_errors-1024x972.jpg){.aligncenter
.size-large .wp-image-1376 loading="lazy" decoding="async" width="520"
height="494"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/excel_errors-1024x972.jpg 1024w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/excel_errors-300x285.jpg 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/excel_errors-768x729.jpg 768w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/excel_errors-50x47.jpg 50w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/excel_errors.jpg 1190w"
sizes="(max-width: 520px) 100vw, 520px"}](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/excel_errors.jpg)

The authors speculate that the correlation they found is due to higher
ranking journals publishing larger gene collections. This explanation,
if correct, would suggest that, on average, error detection in such
journals is at least not superior to that in other journals.

The second study is on the statistical power of cognitive neuroscience
and psychology experiments. The authors
[report](https://biorxiv.org/content/biorxiv/early/2016/08/25/071530.full.pdf)
that statistical power has been declining since the 1960s and that
statistical power is *negatively* correlated with journal rank (i.e., a
reproduction of the work above, with an even worse outcome). Moreover,
the fraction of errors in calculating p-values is positively correlated
with journal rank, both in terms of records and articles (even though I
have to point out that the y-axis does not start from zero!):

[![errors](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/errors-1024x301.png){.aligncenter
.size-large .wp-image-1372 loading="lazy" decoding="async" width="520"
height="153"
srcset="https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/errors-1024x301.png 1024w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/errors-300x88.png 300w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/errors-768x226.png 768w, https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/errors-50x15.png 50w"
sizes="(max-width: 520px) 100vw, 520px"}](https://bjoern.brembs.net/wp/wp-content/uploads/2016/01/errors.png)Thus,
there are at least three additional measures in these articles that
provide additional evidence supporting the interpretation that the
highest ranking journals publish the least reliable science.

**UPDATE II (9/5/2017):** Since the last update, there has been at least
[one additional study](https://www.nicebread.de/p-curving-journals/)
comparing the work in journals with different impact factors. In the
latest work, the authors compared the p-values in two different
psychology journals for signs of p-hacking and other questionable
research practices. Dovetailing the data available so far, the authors
find that the journal with the higher impact factor (5.0) contained more
such indicators, i.e., showed more signs for questionable research
practices than the journal with a lower impact factor (0.8). Apparently,
every new study reveals yet another filed and yet another metric in
which high-ranking journals fail to provide any evidence for their high
rank.

 

**UPDATE III (07/03/2018):** An edited and peer-reviewed version of this
post is now available as a [scholarly journal
article](https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full).


tl;dr:
 
 Data from thousands of non-retracted articles indicate that experiments published in higher-ranking journals are less reliable than those reported in ‘lesser’ journals.

Vox health reporter Julia Belluz has recently covered the reliability of peer-review.

Even without retractions, ‘top’ journals publish the least reliable science

[*This is a slightly edited (amended, essentially) version of* [my
article](https://theconversation.com/open-access-and-the-looming-crisis-in-science-14950)
published today at [The
Conversation](https://theconversation.com).]{style="font-size: 12px;"}

In cases where a problem within a community is detected and collective
action is required to address the problem. one needs to strike a fine
line or any efforts to convince the community that action is required
will fail. If one describes the problem too cautiously, the message will
be ineffective in rallying the community and collective action will not
happen. Describe the problem too drastically, and the community will not
be convinced by the alarmist rhetoric and also fail to take action.

Here are some data, together with a computed trend that indicates what I
think would be best described as a looming crisis for the scientific
community: not yet urgent, but if no action is taken, the trend will
threaten the entire scientific enterprise within the coming 30 years.

Currently, the number of scientific papers retracted from PubMed is a
mere 0.05%, an astonishingly low rate, given the vagaries of the job and
the common .05 statistical significance level in biomedicine. However,
recently this rate has been rising. The rise is exponential such that,
if the trend were to continue, by about 2045 almost all scientific
publications will have to be retracted. Apart from high-profile fraud
cases which since recently also make it into the general news media, the
numbers themselves are not yet alarming. Extrapolating the current
trend, we will not reach the .05-level, i.e. 5% of all published
articles in a given year will be retracted, before 2033 -- it will take
almost twenty years for this hundredfold increase, i.e. doubling about
every three years. If the current trend were to continue that far into
the future, 10% would be reached about 2036, 20% in 2039, 40% in 2042,
80% in 2045 and soon thereafter, for every new publication in PubMed, at
least one older one would have to be retracted.

### Causes

Usually, any problem at this scale will have a number of underlying
causes. Science, like any enterprise, is a complex affair. In this
special case, however, a large part of these causes can be traced to the
reward structure that scientists have built around them to make this
enterprise work. They are related to how scientists choose to publish
their results and how fellow scientists reward this publishing behavior.

Unfortunately, even with a clearly identifiable cause, in this case
there is no easy fix.

### History

Starting in the 1960s, the rising number of scholarly journals increased
subscription costs for libraries too quickly. Institutional libraries
and university faculty were faced with the problem of deciding which
journals to read and subscribe, keeping their budgets in mind. One
solution, it seemed, was to rank journals according to how many
citations each of its publication garnered on average.

Back then only a few realized that a generation later their future
colleagues would not only adjust their reading habits to journal rank,
but that those reading habits will translate into new publishing habits.
After all, every one wants their work to be read by as many as possible.

Today, where you publish matters more than what you publish. This
behavior may vary from country to country and field to field, but it is
quite widely known that tenure, promotion, grant funding, etc. all
depends on where in the hierarchy of journals one's research results
have been published.

What makes it worse is this feedback loop: more citations means more
attention. More attention more citations. As long as space is scarce,
more manuscript submissions mean higher rejection rates, for some
journals the rejection rates can be as high as 90%. Like long lines in
front of hip clubs, higher rejection rates mean more validation for the
few researchers that actually manage to get published in these journals
and hence more citations.

### Publishers

Not surprisingly, as researchers adjusted their behavior, so did
publishers. The top journals did not increase space for the increased
demand. Instead, they spawned ever more lower ranking offspring to
reinforce the ratchet and keep the demand high.

With the arrival of the internet, publishers went through a complete
reversal of their role in knowledge dissemination. Before the internet,
publishers would heed their profession, investing in printing presses
and delivery systems for a wide and efficient system of dissemination to
the public. Today, academic publishers have become the opposite. These
corporations are now "hiders": they invest in paywalls, hiding the
research from the public that paid for it.

But perhaps this is all fine and well? After all, publishers select the
very best science using professional editors and need to protect their
investment.

### Rankings

Superficially, one may assume that such strong selection would only
enable the most groundbreaking, methodologically sound research to clear
such monumental hurdles. However, [the data speak a different
language](https://www.frontiersin.org/Human_Neuroscience/10.3389/fnhum.2013.00291/full).
While indeed quantifications of 'novelty' and 'importance' have been
found to correlate significantly with journal rank, the correlation is
so weak, that for every publication correctly published in a
high-ranking journal, there is one that should not have been published
there. Conversely, such a large percentage of research findings that did
not manage to get published in a top journal end up being recognized as
deserving after being published in a lower journal, that the practical
value of journal rank as an evaluation signal is negligible, despite the
statistics. Even more damaging, however, quality-related metrics such as
technical quality, methodological soundness, reproducibility and so on
either failed to show any correlation with journal rank, or correlate
*negatively*. These data support the observation of decreasing intervals
between prominent scientific fraud scandals and the increasing range and
scope of each case, mostly based on fraudulent work published in top
journals.

### Marketing

Apparently, as the pressure mounts to publish in top journals, authors
race to find a way to present their work as important and
groundbreaking, no matter the actual content and quality. Science today
has become a rat race between authors marketing their research and
professional editors struggling to see through the fluff. It is the
perfect arrangement to bring our the worst in all participants:
scientists fear for their livelihoods and do what is necessary to
survive, with little regard to science. Publishers fear their business
model and do what is necessary to survive, with little regard to
science. And librarians are caught in the middle.

### Multiplicators

This straightforward, evidence-based analysis of the current state of
affairs suggests a slightly more alarmist suspicion. Exponential effects
are typical for social phenomena, where feedback is driving much of the
dynamics of the system. The already described, relatively fast feedback
loop propped up by journal rank supports a second, slower loop: after a
generation of increasing pressure to publish in top journals, many of
the now leading scientists are excellent at marketing their research to
top journals. After all, this is how they were selected. It may of
course happen that these researchers are also excellent scientists, but
this was often enough not the selection criterion. These scientists are
now training the next generation of scientists in how to be successful
in science. The anecdotal evidence suggests that -- intentional or not
-- marketing one's research to top journals may play a dominant role in
this training. If this were indeed the case, it would help explain the
exponentially increasing retraction rates in the scientific literature
and constitute cause for grave concern with respect to the looming
crisis in science.

### Reform

Of course, laziness, vanity, ambition, desperation, hubris,
self-aggrandizement, sleep deprivation or bad time management are not
going to disappear any time soon. Scientists are human too. The
realistic task can thus not be to altogether eliminate shoddy,
unreliable science, but to reform our infrastructure which currently
almost appears to be purposefully designed to exacerbate aforementioned
traits and bring out the worst in scientists, thereby wasting resources
and threatening lives. Scientific discoveries are unreliable enough even
under the best of circumstances, given the complexity of the matter,
there surely is no need to worsen the situation. Instead, we must design
an infrastructure that brings out the best in scientists and aligns
their incentives with that of science and the public.

### Technicalities

Given the current status quo, this is technically rather
straightforward: one has to remove the incentives for scientists that
encourage and reward marketing at the expense of science and instead
encourage and reward actions of scientists that benefit both science and
scientists. Given that scholarly communication is at the heart of the
scientific endeavor, this is where we should place most emphasis in our
infrastructure reform. What we need is to abandon the traditional idea
of journal publishing altogether, in favor of an institution-based
platform that allows every tax-payer, scientist or not, to access all
the literature, data and software that is being paid for out of the
public purse. For the past two decades, corporate publishers have proven
to have become exceedingly poor custodians of the public good that is
research results. From their continued and often shamefully backstabbing
and cunning resistance to change their business model to provide even
simple read-only access to the scientific literature to those who paid
for the research in the first place, to the current
[prohibition](https://www.nature.com/news/tensions-grow-as-data-mining-discussions-fall-apart-1.13130)
of text- and data-mining access for scientists to their own literature,
corporate publishers have proven beyond the shadow of a doubt that they
have forfeited every right to be perceived as partners of scientists.
Instead, the evidence weighs heavy that many if not most of these
corporations are now enemies of science.

### Solutions

Small versions of such open access platforms exist in many university
libraries today, in various forms. They need to be combined and made
interoperable using common standards. In less fortunate parts of the
world, where research billions are not as abundant, governments felt
forced, long before us, to realize how wastefully corporate academic
publishing treats tax-funds. These countries have very successfully been
running a multi-national platform,
[SciELO](https://bjoern.brembs.net/2013/06/cut-out-the-parasitic-middlemen/),
which, if implemented on a global scale and funded with only a single
digit fraction of what corporate publishing costs today, would eliminate
the pernicious incentives currently in place and provide full open
access to the public. Recent initiatives like
[SHARE](https://www.share-project.org/publications/journalarticles0.html),
the [DPLA](https://dp.la/) or the
[LPC](https://www.educopia.org/programs/lpc) could serve as nucleation
points for such developments in the rich countries. Provided that the
reputation system built into this platform is guided by the same
evidence-based reasoning and tested with the same scientific rigor we
commonly apply to our experiments, this kind of reform would not only
come for free, it would save billions in tax-funds every year and, by
virtue of its public accessibility, literally save countless lives.

It is a collective disgrace for the entire scientific community that
saved money and saved lives are apparently not sufficient incentives to
drive forceful and rationally designed reform right now. Instead, as in
many other communities, vested, not exclusively corporate, interests are
still keeping an upper hand.  
This must change.


This is a slightly edited (amended, essentially) version of

my article published today at The Conversation. In cases where a problem within a community is detected and collective action is required to address the problem. one needs to strike a fine line or any efforts to convince the community that action is required will fail.

Publicaciones de Rogue Scholar

How shameful should retraction be?

Even without retractions, ‘top’ journals publish the least reliable science

The Conversation: The looming crisis in science