Q&A with Berna Devezer

Dr Devezer is a Professor of Marketing, affiliate faculty at the Department of Mathematics and Statistical Science, and Assistant Director at the Institute for Modeling Collaboration and Innovation (IMCI) at the University of Idaho. She holds a PhD in Marketing and an M.S. in Statistics from Washington State University.

She is interested in advancing models and theories of the science of science. Her research brings together statistical theory, experimental methodologies, mathematical modeling, philosophy and history of science to advance an interdisciplinary understanding of the scientific process.

(1) What first got you interested in this area?

My own experiences in experimental consumer psychology during grad school and beyond. I ran so many experiments that did not work, so many replications of published experiments that failed. And I experienced so much frustration with the content of the graduate methods courses I had to take, so much self-doubt about my own perception of the quality of research getting published in our top journals. When the initial conversations about replications, p-hacking, and research quality started to pick up speed, I was immediately on board because I had so many questions of my own. Turned out my vision did not quite align with that of the science reform movement, and my research program diverged, but the fact that we were having these conversations gave me the initial push I needed pre-tenure to start formally asking the questions that were keeping me awake at night.

(2) What's your biggest concern about mainstream metascience?

It's difficult to rank because I have many, but my major concerns could be boiled down to one thing: let's call it a "cart before the horse" attitude. Many metascientific claims and prescriptions about science have preceded scholarly research, and these unexamined conclusions continue to guide most of metaresearch. This is also where my own path diverged from the mainstream, because I was curious and wanted to understand the what, how, and why of scientific progress (or lack thereof) rather than believing pre-packaged answers science reform pioneers provided. Mainstream metascience has completely bought into its own hype and lost its ability to ask consequential questions and genuinely seek answers.

(3) What's your take on the replication crisis?

It's past time we stopped using the phrase altogether. It's not only misguided but has started to become actively harmful to the scientific endeavor and community. The first part (misguidedness) of my sentiment is based on my own research. After a decade of studying replications and reproducibility, it would be self-defeating if I still believed that there was a replication crisis or that the label itself made any sense. For the specifics, I can refer the blog readers to my research program, starting with our 2019 PLOS ONE paper.

The second part (harms) is due to the ongoing assault on science and higher education, and external interventions in both, in the US and beyond. In the current political environment, the crisis rhetoric is being routinely weaponized as evidence that science is broken and scientists can't be trusted. I recommend this recent post by Jeff Lockhart as an insightful read.

(4) How do you view open science?

It depends on what is included under the umbrella. There are many definitions and even more interpretations. In general, openness, transparency, and honesty are important and uncontroversial scientific virtues that we should cultivate and strive to uphold. But a virtue is not a standard. I think every field needs to determine the boundaries of what open science means within that context, carefully evaluate who bears the costs and what potential harms need to be mitigated. Coming up with universal rules, checklists, or impositions that apply to all of science is futile, and essentially authoritarian. I am also not a fan of treating arbitrary open science criteria as a purity test. Quick and intentionally broad answer because I don't think it's mine to settle: open science is an open discussion about how we can open the boundaries and gates of science to internal and external audiences.

(5) What's one thing you'd change about the way we do research?

Not quite about how we do research but something more systemic: take the prestige economy and citation metrics out of the equation altogether. Perhaps I'd want to create more resources, distribute them more equitably, take publication pressures away, and provide the support people need to do the best research they can. But I fear all of that might require a magic wand, and if I had it, I would hesitate to use it. It's not that I don't believe we can change the current system, but in all likelihood, my well-intentioned attempt would end up replacing it with something problematic in different ways; perhaps a new kind of hierarchy that's just as bad if not worse. And I am not being cynical. I just do not trust that external, top-down interventions in complex social systems have straightforward, positive consequences. I'd rather change how I do research (as I have).

(6) Which one of your papers should we read first and why?

Our first metascience paper in PLOS ONE (mentioned earlier) Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity because I genuinely believe it is fantastic. It also exemplifies how my path diverged from mainstream metascience. The modeling approach is very novel and thorough. It's exploratory, interdisciplinary, ambitious yet humble, insightful, and rigorous. If we had a certain type of street/career smarts, I'm sure we could have turned it into five different papers or sent it to a glamor journal. We were all ECRs, simply extremely excited about the research and the collaboration, had open minds and no assumptions about what we should find. We did not treat modeling as a vehicle to get to the answers we needed but as a challenge that compelled us. So the paper is curiosity-driven and is based on years of hard work. I love what we did every time I reread it.

(7) What's a common misconception about your work you'd like to correct?

I don't have a good understanding of how my work is perceived to be able to identify misconceptions. I do know how it gets cited, however, and it looks like most colleagues don't read past the abstract or the conclusion. Just wish more readers were interested in the modeling and the theoretical contributions as ends in themselves instead of citing (or criticizing) the work for soundbites. Most reviews and comments we get critique the implications of our conclusions as if that's something we should have planned for before doing the research. It's all backwards. Some readers may disagree with the conclusions that follow from our work, but that's what we found, and pushing back is not going to change the evidence. Let's focus on what we can learn from it.

(8) What are you working on at the moment?

Currently finalizing our latest work The difference between "replicable" and "not replicable" is not itself scientifically replicable before submission. Luckily, we got some of the good (genuinely useful) kind of informal feedback on the preprint, raising valid questions about our model. It's helped us clarify some misunderstandings, improve a few weaknesses, and hopefully present the theory more clearly. After that, I hope to move away from replications and reproducibility for a while since we have done quite a lot in this domain and made a lot of theoretical progress. Hope to revisit our original model in the PLOS ONE paper mentioned earlier and expand it to study other aspects of the scientific process. I have a lot of theoretical questions, all of which require translating theory and philosophy to math, which requires a lot of brain juice.

(9) What advice do you have for researchers starting out in your area?

I'm not good with advice. Since I don't like taking it, it's only fair that I don't offer it either. Not that I have an area to speak of. What do I call it? Theoretical metascience? Still a long way from becoming an area.

(10) What question should I have asked you, and what's your answer?

Q: In your research program, what has benefited you most?

A: Reading outside of my area (such as it is) and collaborating with colleagues from diverse backgrounds and disciplines. In the last decade, I have read anything I found interesting, setting all pragmatic considerations and disciplinary boundaries aside. I find myself reading books and papers in statistics, mathematics, philosophy of science, history of science, science and technology studies, psychology, sociology, computer science, economics, political science — you name it. Qualitative, quantitative, empirical, theoretical… I was also lucky enough to interact and collaborate with scholars from many fields. I cannot imagine doing the work of the past decade without such cross-pollination. Allowing myself to feel professionally homeless (without a proper base discipline) has set me free to do the kind of work that compels and excites me, and that I find internally rewarding.

Subscribe now

Share