Ingo Rohlfing

*Is Open Science passé?* is the question asked by Xenia Schmalz [in this
blogpost](https://xeniaschmalz.blogspot.com/2024/04/is-open-science-passe.html).
I recommend reading it before I share brief thoughts on some points that
are raised.

I wish an open science movement was not needed anymore, but I agree this
is most likely not the answer to the leading question. Neither has the
open science movement failed; progress toward more transparent and
credible science is simply slow. When one looks at [how many articles
share code and
data](https://replicationnetwork.com/2024/04/10/aoi-mass-reproducibility-and-replicability-a-new-hope-by-brodeur-et-al-2024/)
(or, maybe, how many do not); [what the quality of the shared material
is](https://doi.org/10.1038/s41597-022-01143-6) (README file, code
annotation etc.); how often [substantive significance is not discussed
or misinterpreted](https://doi.org/10.1093/esr/jcw047) in quantitative
research etc., I feel there is still much room for improvement. The
question is what this means for open science and meta science.

Does it make sense to write another article on the misinterpretation of
the p-value in a given fieldor subfield? Probably yes, it depends on the
field. It may be less needed in a field like psychology because there
seems to be a larger degree of awareness to matters of research quality
and credibility. In other fields, probably including political science,
it may be interesting to review the interpretation of p-values and
substantive significance because fewer graduate students may have
received training on this. Do we need another many-analysts study
showing that results differ across analysts? I am less sure about that,
but this is a topic for a separate blog post.  

I think the reviewer comment that motivates the blog post by Xenia
Schmaltz illustrates that for many papers the study and results may be
obvious to a researcher from the field, leading to a comment like
"*What's the news here?*". For many other researchers, however, the
study may be interesting and important to read because they do not
follow closely work on open science and meta science. It is fine when an
empirical researchers pays most attention to theories and empirical
issues that are pertinent to the own subfield because this is where the
primary interest lies. For open science and meta science reserchers,
this means that there is still much work to do, including research,
teaching and publishing on topics that may seem like old news to experts
from these two fields.


<em>
 Is Open Science passé?
</em>
is the question asked by Xenia Schmalz in this blogpost. I recommend reading it before I share brief thoughts on some points that are raised. I wish an open science movement was not needed anymore, but I agree this is most likely not the answer to the leading question. Neither has the open science movement failed; progress toward more transparent and credible science is simply slow.

Open Science is neither passé nor is it there yet

In place of a generic blog post, I am reposting a short Twitter thread
here. The thread is a response to an opinion piece on the Times Higher
Education website titled [Pay researchers for results, not
plans](https://www.timeshighereducation.com/opinion/pay-researchers-results-not-plans){rel="noreferrer noopener"
target="_blank"}. (Posts on the THE website require registration of an
account that includes a couple of free reads.)

I copy-paste thread into this post. If you prefer to read it on
Threadreader, you find it
[here](https://threadreaderapp.com/thread/1502654201306009612){target="_blank"
rel="noreferrer noopener"}.

------------------------------------------------------------------------

Lots of arguments in
<https://www.timeshighereducation.com/opinion/pay-researchers-results-not-plans>
I disagree with or are questionable. To start with a point of agreement:
Yes, current modes for getting competitive research grants are not
ideal. But I am fairly skeptical the proposed alternative is superior 1/

Let's leave aside that 'impact' seems to be equated with publishing and
citations: Getting paid for publications would, in the current system,
most likely not improve research and funding allocation. I think it
wouldn't be different than it is right now where researchers 2/

aim to get published in journals that are perceived to be of high
impact/high quality bc of potential positive effects for grant
applications, tenure, promotions etc. Funding agencies paying for
publications in a list of "approved venues" (who would decide about this
list?) 3/

would set strong and potentially wrong incentives to get published in
these venues (p-hacking etc.).  
Another proposal is that authors who published in these venues can give
money to authors they cited in their own study. The idea is that authors
can reward 4/

"major enablers or precursors" of the new research. It is difficult to
see how this could not contribute to formation of clubs, networks and
collusive behavior to keep the money inside your own research clique.  
The claim that science is an industry is to some degree right, 5/

but the conclusion is not, IMO. An analogy to industry is: "You buy a
car, not a document describing a car that may or may not be built, and
the carmaker gets paid when it delivers the car." First, I guess in the
history of car-making there had been funding for plans for cars 6/

that were not built. Second, this conflates basic and applied research,
where the article seems to have a strong preference for the latter.
Third, a car-buyer knows that the car meets certain standards bc of laws
and regulations that are enforced (most of the time, at least) 7/

This is not where I see scientific publications (generally speaking) at
the moment wrt to transparency, reproducibility (if applicable) etc.
Funding allocation could be partially revised by attaching higher
priority to these elements of research. 8/

It is then of secondary importance whether a single plan to build a car
worked or didn't work and whether there is a car or not. If certain
standards are followed and enforced, one can be confident that the
result is credible and can serve as the basis for follow-up research 9/


In place of a generic blog post, I am reposting a short Twitter thread here. The thread is a response to an opinion piece on the Times Higher Education website titled Pay researchers for results, not plans. (Posts on the THE website require registration of an account that includes a couple of free reads.)  I copy-paste thread into this post. If you prefer to read it on Threadreader, you find it here.

Is payment for publications and paying for citations a better to way to fund research? Most likely not

Some ideas about how the peer review process in academic publishing can be used as a device for preregistering parts of an empirical analysis.

Preregistration of an analysis requires the prior unobservability of
data. One publicly declares what one intends to do before one can access
data. One goal of preregistration is to [diffuse potential concerns that
a publication misrepresents the order in which an empirical analysis was
implemented](http://cega.berkeley.edu/assets/cega_events/45/HumphreysFishingPolAnal.pdf){target="_blank"
rel="noreferrer noopener"}. Reasons for prior unobservability of data
could be that the data has yet to be produced (as in experiments and
[some observational
designs](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2731920){target="_blank"
rel="noreferrer noopener"}), or that access to the data is restricted
(such as sensitive microdata that can only be accessed locally and by
certified approval of the data host). Based on a peer review experience
a friend of mine and I had, it occurred to me that there is a second way
of preregistration that works *regardless* of the prior observability of
data.

In our peer review process, a reviewer suggested to additionally test
our argument with different data than we had used. We had not thought
about using the data before and did it at the reviewer's suggestion. (I
don't remember whether we had known it; I don't think so.) The good news
for us was that an analysis with the alternative data produced evidence
that confirmed our original results. This confirmation is qualitatively
different from the results that we reported in the first submission and
it should count more, in my opinion.

The reason is that the peer review process and the reviewer comments in
particular worked as a preregistration device. The data the reviewer
recommended to use was available to us, but we weren't aware of it and
if we had been, we did not use it. What was unavailable to us was the
idea of using the data in the way proposed by the reviewer. In analogue
to the idea of 'prior unobservability of data' that underlies standard
preregistration, one could call the experience that we made *prior
unavailability of an idea* (or *prior lack of imagination* or so).

The advantage of preregistration through peer review is that it is
available regardless of the type of design -- experimental or
observational -- and the type of method -- quantitative or qualitative.
For example, imagine a researcher has done a qualitative content
analysis of publicly available speeches of party leaders. The original
analysis cannot be preregistered because the speeches have been publicly
available from the start of the project. Now, the peer review process
would work as a preregistration device if a reviewer (or an editor) has
an idea for a coding category that the researcher had not thought of.
When the researcher goes back to the speeches and codes them based on
the new category, this part of the analysis would be preregistered. The
peer review process allows one to timestamp the origin of the idea in
terms of the time when it was proposed and the person who proposed it
(the originator).

There are several requirements that have to be met for effective
preregistration through peer review, where 'effective' means that the
researcher's degrees of freedom in misrepresenting an analysis are
minimized. First, the peer review process needs to be made
*transparent*. The reader of an article needs to know who the originator
of an idea is -- the author or a reviewer -- and when the idea was
introduced by the originator. A published manuscript would need to
specify that an element of the empirical analysis was proposed by a
reviewer and that this part of the analysis was only implemented after
the reviewer proposed to do it.

Second, this information would not be enough because the reader needs to
know *how specific* the reviewer's proposal was. The more specific the
reviewer's comment, the smaller the researcher's degrees of freedom when
implementing it and the fewer opportunities to misrepresent the analysis
and "manufacture" results such that they support a preferred theoretical
interpretation.

Third, the reader needs to know what a reviewer proposed to do and what
an author *declined* to implement. With this information, the reader
would not be able to tell whether an author refused to follow the
reviewer's recommendation because of the belief that it is superfluous
or flawed in terms of substance, theory or method; or because the author
did not want to include it because the proposed analysis did produce
results that challenge the conclusion in the manuscript and were
withheld as a consequence of this. Still, it is useful to make
transparent where reviewers and authors disagreed; keeping in mind that
even many declined reviewer requests are not a strong indication of
selective reporting because the reviews could make ill guided
recommendations. The degree of disagreement would be an indication of
how selective the author was in following the reviewer's comments and
could be a guide to a follow-up technical replication of the analysis.

Altogether, this shows that preregistration through the peer review
process requires transparent peer review that makes public all versions
of the manuscript -- the first submission plus all revisions -- and all
reviewer reports, which could be anonymized for preregistration
purposes, and editorial letters.


Preregistration of an analysis requires the prior unobservability of data. One publicly declares what one intends to do before one can access data. One goal of preregistration is to diffuse potential concerns that a publication misrepresents the order in which an empirical analysis was implemented.

Messaggi di Rogue Scholar

Open Science is neither passé nor is it there yet

Is payment for publications and paying for citations a better to way to fund research? Most likely not

The peer review process as a preregistration device