Funding and evaluating dynamic vs static research products

(please cite this post as https://doi.org/10.59350/nxkfm-ghk92)

Today many researchers are not only capturing new knowledge in papers; they are also creating elements of infrastructure, such as datasets, software, and services. However, traditional methods of support and evaluation haven’t made a concomitant change. This short blog post is not intended to comprehensively address the specific challenges of support and evaluation, which I and others have been working on for many years, but I hope it is a brief introduction (or perhaps a reminder) of why such a change is needed.

In the “traditional” model, a funder might support a researcher to perform research that leads to a paper. The knowledge captured in this paper is static (fixed once the paper was published), and it is transmitted to others via that paper, without the researcher playing an active role. Or perhaps it is transmitted by a person, such as a student who was involved and then went elsewhere in a postdoc, staff, or faculty role. For a particular item of knowledge, the creator’s role ends when the paper is published, though of course, they likely continue research in the same area, further building new knowledge. And the researchers can be evaluated on those products, and perhaps on the impact they have, with the researcher’s further role being one of mild advertising of the results, for example, via talks and social and other media.

Datasets, particularly static datasets that are collected and deposited in a repository, are similar, in that a researcher (or team) collects and “publishes” them, and then is evaluated on having done so, and perhaps on their usage, which mostly happens without the researcher’s involvement, other than again by publicizing the dataset.

Software and services are quite different from papers in many ways. The main difference is that they aren’t ever “done”, in general. Unless they are quite simple, they need ongoing support and maintenance to be useful. This means that the researcher (or a different set of people) needs to continue to work on the software, preventing software collapse since the entire software ecosystem is dynamic, and addressing bugs, adding features, and providing support to users. 

This ongoing work needs to be supported, but research funders typically make short awards to create knowledge, not to support systems, and even when they do support systems, they typically do so for short periods, without having a long-term view or plan for the specific pieces of software. 

Evaluation for these software projects and systems is also different than for papers, as their impact is much harder to measure. Software can be incorporated in other software via copying or via means such as libraries, which are not captured in traditional metrics. The impact of the software or system is also based in part on how well it is supported and how it evolves, not just what it was like at a single point in time. 

In my opinion, most stakeholders involved in research funding and evaluation have focused on challenges related to the environment around research products, and haven’t internalized that the changes in the products themselves that have occurred as we’ve moved beyond just static papers is a strong driver of the need for changes in such funding and evaluation.