Critical perspectives in metascience and the philosophy of science relating to open science, science reform, and the replication crisis.

Mark Rubin

<p><a href="https://www.researchgate.net/profile/Thomas-Hostler-2">Tom Hostler</a> has just published an article titled &#8220;<a href="https://doi.org/10.36850/mr5">The invisible workload of open research</a>&#8221; in the <em>Journal of Trial and Error&#8217;s</em> special issue on the &#8220;<a href="https://journal.trialanderror.org/special-issue-consequences-of-the-scientific-reform-movement">Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction</a>?&#8221; It&#8217;s an insightful and thought-provoking piece that uncovers the potential workload costs of open research, considers why these costs may be ignored by university management, and makes some useful recommendations for moving forward. Here&#8217;s a few quotes to whet your appetite.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://doi.org/10.36850/mr5" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png" width="1333" height="796" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png","fullscreen":null,"imageSize":null,"height":796,"width":1333,"resizeWidth":null,"bytes":184638,"alt":null,"title":null,"type":"image/png","href":"https://doi.org/10.36850/mr5","belowTheFold":false,"topImage":true,"internalRedirect":null}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45026131-ae6b-4dac-8fbf-6f228053a4d9_1333x796.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></a></figure></div><h4> <strong>The Problem</strong></h4><blockquote><p>The benefits of open research practices have been widely discussed (Munaf&#242; et al., 2017), and generally speaking researchers have positive attitudes towards adopting open research practices, finding them worthwhile (Eynden et al., 2016; Lowndes et al., 2017), and recommending their use to others (Sarafoglou et al., 2022). However, high time cost is repeatedly identified as one of the main barriers to the adoption of open practices (Eynden et al., 2016; Gownaris et al., 2022; Tenopir et al., 2011) and time is the main thing that researchers lack, with workloads at capacity across the sector, having already reached &#8220;untenable&#8221; levels (Long et al., 2020; Papadopoulos, 2017).</p></blockquote><p>In short, implementing open science reforms take work, and work takes time, which academics have very little of! </p><h4><strong>University Management are Not Motivated to Help!</strong></h4><blockquote><p>The theory of academic capitalism suggests that the responsibility for addressing potential discrepancies between modelled and actual workload will not be taken up voluntarily by university management, who may at best ignore such issues, or at worse tacitly approve of them as a form of capitalist efficiency (Lyons & Ingersoll, 2010). In other words, if university management can choose not to incorporate open research into workload models, then they won&#8217;t.</p></blockquote><p>I don&#8217;t think many working academics need to refer to the theory of academic capitalism to appreciate this point!  But to quell any doubt, Hostler refers to empirical work by Kenny and Fluck (<a href="https://doi.org/10.1007/s10734-021-00747-y">2022</a>) and Miller (<a href="https://doi.org/10.1080/1360080X.2019.1635328">2019</a>). For example, Kenny and Fluck (2022) found that:</p><blockquote><p>Management knows that academics will do research (and other work) in their own time and exploit that&#8230;.What is written on paper in terms of hours re admin or research does not correspond to the actual time spent. It all looks fine on paper but has enormous hidden workloads. </p></blockquote><p>Hence, Hostler concludes:</p><blockquote><p>There is a high potential for expectations to engage in open research to become widespread, but without additional time on academics&#8217; workloads dedicated to the activity. </p></blockquote><h4><strong>Some Open Research Advocates Do Not Acknowledge Open Research Workload</strong></h4><blockquote><p>Suggestions that concerns about the workload of open research are an example of a &#8220;myth&#8221; (Bastiaansen, 2019) or a &#8220;misconception&#8221; that can be corrected by &#8220;positive advocacy&#8221; (Hagger, 2022) are unhelpful to having honest conversations about the practical negative consequences of conducting open research.</p></blockquote><p>This is an important point. Some open science advocates engage in a pretty hard sell, and this can include omitting or downplaying some of the costs of open science. However, like Hostler says, it&#8217;s worth having &#8220;honest conversations&#8221; about the pros and cons of open science in order to set more realistic expectations.</p><h4><strong>A Key Recommendation</strong></h4><p>I&#8217;ve just picked out one of Hostler&#8217;s recommendations below to keep things brief, but note that he makes several more in his article:</p><blockquote><p>When promoting open research reforms in universities or conducting open practices themselves [open research advocates] should advocate not only for investments in open research infrastructure and training but to receive extra time in workload allocations to acknowledge the additional burden of open research.</p></blockquote><p>Getting open research work acknowledged in university workload models is likely to be easier said than done! As Hostler explains, open research work often consists of many small tasks, rather than a few large tasks, and so it&#8217;s easily overlooked in workload models. Nonetheless, cumulatively, open research workload can be substantial and, as Hostler notes, if it&#8217;s ignored, we risk &#8220;death by a thousand 10-minute tasks&#8221; (Bozeman et al., 2021).</p><div class="captioned-image-container"><figure><a class="image-link is-viewable-img image2" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg" width="583" height="500" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg","fullscreen":null,"imageSize":null,"height":500,"width":583,"resizeWidth":null,"bytes":75314,"alt":null,"title":null,"type":"image/jpeg","href":null,"belowTheFold":true,"topImage":false,"internalRedirect":null}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F926ef13c-9658-42d0-a89f-5ea0abfb214f_583x500.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="#FFFFFF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></a></figure></div><h4>The Article</h4><p>Hostler, T. J. (2023). The invisible workload of open research. <em>Journal of Trial & Error.</em> <a href="https://doi.org/10.36850/mr5">https://doi.org/10.36850/mr5</a>&nbsp;</p><div class="subscription-widget-wrap" data-attrs="{"url":"https://markrubin.substack.com/subscribe?","text":"Subscribe"}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Critical Metascience ! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>

Open Science and Overwork

Opening Up Open Science to Epistemic Pluralism

<p>In this <a href="https://doi.org/10.36850/mr4">new article</a>, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that&#8217;s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. My aim is not to cast aspersions on the field of metascience but to encourage a deeper consideration of its more questionable research practices, assumptions, and perspectives.</p><div class="captioned-image-container"><figure><a class="image-link is-viewable-img image2" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg" width="518" height="388.8888888888889" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/a5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg","fullscreen":null,"imageSize":null,"height":500,"width":666,"resizeWidth":518,"bytes":51259,"alt":null,"title":null,"type":"image/jpeg","href":null,"belowTheFold":false,"topImage":true,"internalRedirect":null}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa5f016ff-5378-4580-8535-a4bcd097b2ef_666x500.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="#FFFFFF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></a></figure></div><div><hr></div><h3>10 QMPs</h3><ol><li><p><strong>Rejecting or ignoring self-criticism</strong>: Rejecting or ignoring criticisms of metascience and/or science reform</p></li><li><p><strong>Fast &#8216;n&#8217; bropen scientific criticism</strong>: A quick, superficial, dismissive, and/or mocking style of scientific criticism</p></li><li><p><strong>Overplaying the role of replication</strong>: Assuming that replication is essential to science, and that it indexes &#8220;the truth&#8221;</p></li><li><p><strong>Unspecified replication rate targets</strong>: Assuming a replication rate is &#8220;too low&#8221; without specifying an &#8220;acceptable&#8221; rate</p></li><li><p><strong>Metabias</strong>: An unacknowledged bias towards explaining the replication crisis in terms of researcher bias</p></li><li><p><strong>A bias reduction assumption</strong>: Focusing on selective reporting as the primary form of researcher bias and assuming that it can be reduced without increasing other forms of bias</p></li><li><p><strong>Devaluing exploratory results</strong>: Devaluing an exploratory result as being more &#8220;tentative&#8221; than a confirmatory result without considering other relevant issues (e.g., quality of associated theory, methods, analyses, transparency)</p></li><li><p><strong>Presuming QRPs are problematic</strong>: Presuming that <em>questionable </em>research practices (e.g., HARKing) are always <em>problematic </em>research practices</p></li><li><p><strong>A focus on knowledge accumulation</strong>: Conceiving knowledge accumulation as the primary objective of science without considering (a) the role of specified ignorance or (b) different objectives in other philosophies of science</p></li><li><p><strong>Homogenizing science</strong>: Focusing on specific approaches (e.g., quantitative methods; replicable effects) as &#8220;the scientific method&#8221;</p></li></ol><div><hr></div><h3>A Caveat</h3><p>In my article, I stress that only <em>some </em>metascientists engage in <em>some </em>QMPs <em>some </em>of the time, and that these QMPs may not <em>always </em>be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, I think that QMPs should be viewed as invitations to ask questions about how we go about doing better metascience.</p><div class="captioned-image-container"><figure><a class="image-link is-viewable-img image2" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg" width="390" height="471.9" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/d160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg","fullscreen":null,"imageSize":null,"height":605,"width":500,"resizeWidth":390,"bytes":83766,"alt":null,"title":null,"type":"image/jpeg","href":null,"belowTheFold":true,"topImage":false,"internalRedirect":null}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd160ceec-8493-446f-9352-de9bd3860f4d_500x605.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="#FFFFFF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></a></figure></div><div><hr></div><h3>The Article</h3><p>Rubin, M. (2023). Questionable metascience practices. <em>Journal of Trial and Error. </em><a href="https://doi.org/10.36850/mr4">https://doi.org/10.36850/mr4</a></p><h3>Journal of Trial and Error Special Issue</h3><p>My article is part of a special issue in the <em>Journal of Trial and Error</em>: &#8220;Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction? <a href="https://journal.trialanderror.org/pub/callscientificreform/release/4">https://journal.trialanderror.org/pub/callscientificreform/release/4  </a> </p><h3>List of Critical Metascience Articles</h3><p>Many of the articles that I cite in my article are included in this chronological list of critical metascience articles: <a href="https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-critical-metascience-articles">https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-critical-metascience-articles</a></p><div><hr></div><p class="button-wrapper" data-attrs="{"url":"https://markrubin.substack.com/subscribe?","text":"Subscribe now","action":null,"class":null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://markrubin.substack.com/subscribe?"><span>Subscribe now</span></a></p><p></p>

In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that&#8217;s been...

Questionable Metascience Practices

<p>Sabina Leonelli&#8217;s new book &#8211; &#8220;Philosophy of Open Science&#8221; &#8211; will be published later this year. However, there is an open access preprint available on the PhilSci Archive <a href="http://philsci-archive.pitt.edu/21986/">here</a>.</p><div class="captioned-image-container"><figure><a class="image-link is-viewable-img image2" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png" width="186" height="280.2567567567568" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png","fullscreen":null,"imageSize":null,"height":446,"width":296,"resizeWidth":186,"bytes":224190,"alt":null,"title":null,"type":"image/png","href":null,"belowTheFold":false,"topImage":true,"internalRedirect":null}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F086be164-e4c0-4dbe-8044-4c0f49f60696_296x446.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="#FFFFFF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></a></figure></div><p>In the book, Leonelli provides &#8220;a constructively critical reading&#8221; of the standard approach to open science which, she argues, is focused on sharing &#8220;objects&#8221; such as data and materials. In Leonelli&#8217;s view, this object-sharing approach has become an obstacle to promoting open science. Grounded in philosophies of science from Popper, Chang, Longino, and others, she puts forward an alternative, process-oriented view of open science that refers to efforts to establish &#8220;judicious connections among systems of practice.&#8221;</p><blockquote><p>I propose a conception of openness as <em>judicious connection</em>, which is grounded in a <em>process-oriented epistemology of science</em> that recognises the situated, embodied and goal-directed nature of communication and collaboration among researchers.</p></blockquote><p>What does &#8220;connection&#8221; have to do with openness?</p><blockquote><p>The idea of openness is quintessentially linked to that of learning as going beyond one&#8217;s boundaries. This arguably applies to systems of research practice as much as to research groups and individual learners: establishing new connections often means expanding one&#8217;s learning, challenging existing assumptions around what is considered external or irrelevant to a given system, and considering whether new boundaries need to be established that incorporate the novel relations.</p></blockquote><p>Importantly, Leonelli does not equate an indiscriminate increase in scientific connections with an increase in open science. Instead,</p><blockquote><p>the building and maintenance of connections need to be judicious: they require skilled deliberation, whereby the new opportunities offered by the connection in question are evaluated within the contexts at hand. Indeed, openness can itself be understood as a dynamic and highly situated mode of valuing the research process and its outputs, which encompasses economic as well as scientific, cultural, political, ethical and social considerations (Levin and Leonelli 2017).</p></blockquote><p>The differences between the object-sharing and &#8220;judicious connection&#8221; approaches to open science also have implications for inclusion. The standard object-sharing approach assumes that greater transparency leads to a more inclusive science. In contrast, Leonelli&#8217;s &#8220;judicious connection&#8221; approach flips the causal arrow and assumes that more inclusion leads to greater transparency.</p><blockquote><p>The implementation of OS [open science] needs to start from consideration of what it may take to make research more inclusive, diverse and just &#8211; rather than expecting such an outcome to naturally follow from the &#8216;right&#8217; choice of software, infrastructures, standards, publishing platforms, or whichever other technological or institutional fix is being devised to facilitate access to resources.</p></blockquote><p>Finally, Leonelli&#8217;s distinction between the two approaches to open science is important because she believes that the standard, object-sharing approach:</p><blockquote><p>threatens to blindly privilege specific ways of knowing, thus potentially disrupting sophisticated methodologies, inadvertently dismissing well-established research traditions, and exacerbating the already large epistemic and social divides separating research domains and locations As denounced by a number of critics in science and science studies, there is a substantive risk of some OS policies &#8211; despite their good intentions and progressive slant - acting as a reactionary force which reinforces conservatism, discrimination, commodification and inequality in research, thus ultimately closing down opportunities for inquiry in a disastrous reversal of what they set out to achieve.</p></blockquote><p>I won&#8217;t go any further into the distinction between the object-sharing and judicous connection approaches to open science. Leonelli already does an excellent job of that in her book. Instead, I want to highlight five quotes that are critical of the standard object-sharing approach, because these are motivating towards a consideration of her alternative judicious connection approach.</p><h4><strong>(1) Standard Open Science Privileges a Homogeneous View of Science</strong></h4><blockquote><p>Many of the more institutionalised OS initiatives tend to privilege a homogenous, universally applicable understanding of the scientific method over a pluralistic and situated one.</p></blockquote><p>Similar points have been made by several others. For example, <a href="https://doi.org/10.1177/10892680221083876">Malich and Rehmann-Sutter (2022)</a> pointed out that:</p><blockquote><p>Metascience often construes science and psychology from a homogenizing perspective that omits the plurality and complexity of both scientific methods and psychological approaches. This omission entails several epistemic risks because the reforms that metascience initiates or accompanies can reinforce these homogenizing tendencies</p></blockquote><h4><strong>(2) Open Science Should Avoid Mindless Transparency</strong></h4><blockquote><p>Open Methods is not a matter of recording and sharing every detail of a research procedure, but rather a reflection on which research components and techniques are most salient to the outcomes, and should thus be accessible and reproducible.</p></blockquote><p>I totally agree. As I argue near the end of this <a href="https://www.youtube.com/watch?v=xsEoLhQrKNQ&t=1s">presentation</a>, I&#8217;m opposed to what I call &#8220;mindless transparency.&#8221;</p><div class="captioned-image-container"><figure><a class="image-link is-viewable-img image2" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png" width="440" height="330.1002734731085" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png","fullscreen":null,"imageSize":null,"height":823,"width":1097,"resizeWidth":440,"bytes":71445,"alt":null,"title":null,"type":"image/png","href":null,"belowTheFold":true,"topImage":false,"internalRedirect":null}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9c63ef6d-9449-4b2d-bcff-9054ced4f8b8_1097x823.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="#FFFFFF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></a></figure></div><p>By the way, I think <a href="https://www.routledge.com/Conjectures-and-Refutations-The-Growth-of-Scientific-Knowledge/Popper/p/book/9780415285940">Popper (2002, p. 230</a>) would also agree with this point: </p><blockquote><p>When the judge tells a witness that he should speak 'The truth, the <em>whole truth</em>, and nothing but the truth', then what he looks for is as much of the <em>relevant truth</em> as the witness may be able to offer. A witness who likes to wander off into irrelevancies is unsatisfactory as a witness, even though these irrelevancies may be truisms, and thus part of 'the whole truth'.</p></blockquote><h4><strong>(3) Reproducibility Should Not be an Unconditional Demarcation Strategy</strong></h4><blockquote><p>Taking a monolithic understanding of reproducibility as a demarcation strategy for the whole of science side-steps the precious plurality of methods developed to suit specific goals, concepts and target objects.</p></blockquote><p>Again, this point has been made by several others (e.g., Devezer et al., 2019, 2021; Feest, 2019; Haig, 2022). Nonetheless, replication is often portrayed as a &#8220;defining&#8221; or &#8220;essential&#8221; aspect of science.</p><h4><strong>(4) Skepticism Against Qualitative Research is Unwarranted</strong></h4><blockquote><p>A case in point is the skepticism towards qualitative research often displayed within debates on reproducibility, which sometimes portray qualitative methods as hopelessly subjective and devoid of rigorous forms of data collection and verification....Assuming that qualitative research traditions in anthropology or sociology are incapable of rigorous research means drastically reducing the diversity of systems regarded as exemplars of good practice.</p></blockquote><p>Yes! We need to be very careful that open science does not reopen &#8220;&#8216;sectarian&#8217; divides between quantitative and qualitative&#8221; research, as <a href="https://blogs.lse.ac.uk/impactofsocialsciences/2022/06/27/three-false-starts-on-the-road-to-open-social-science/">Dunleavy (2022)</a> has cautioned.</p><h4><strong>(5) Open Science Tech and Procedures Should Consider Diversity and Inclusion</strong></h4><blockquote><p>To date, some parts of the OS movement &#8211; particularly its institutionalized, top-down incarnations - have paid too much attention to designing procedures and technologies for sharing, and this has come at the expense of strategies, training and procedures to assess who is included and excluded from such apparatus, understand why and with which implications, and mitigate eventual instances of epistemic injustice.</p></blockquote><p>Another important point! A good example of this lack of attention to inclusion comes from <a href="https://doi.org/10.1111/bjso.12576">Prosser et al. (2022</a>) in their article &#8220;When open data closes the door.&#8221; These researchers found surveyed 261 social psychology journals and found that journal guidelines for sharing data were exclusionist with regards to qualitative data:</p><blockquote><p>Guidelines explicitly addressing open qualitative data were very rare, provided by just three journals (2.1% of all relevant journals). All other journals that made reference to 'open data' did so without making a distinction between qualitative and quantitative data.&#8221;</p></blockquote><h3><strong>Further Information</strong></h3><p>Leonelli, S. (2023). Philosophy of open science. [Preprint] <a href="http://philsci-archive.pitt.edu/21986/">http://philsci-archive.pitt.edu/21986/</a></p><p>Order the published print version <a href="https://www.cambridge.org/core/elements/abs/philosophy-of-open-science/0D049ECF635F3B676C03C6868873E406">here</a>.</p><p><a href="https://opensciencestudies.eu/">The University of Exeter&#8217;s PHIL_OS project (2021&#8211;2025)</a>: &#8220;Aims to develop an empirically grounded philosophy of Open Science [OS] that emphasises the diversity of research environments around the world and articulates the conditions under which OS can leverage such diversity to promote good research practice.</p><div class="subscription-widget-wrap" data-attrs="{"url":"https://markrubin.substack.com/subscribe?","text":"Subscribe"}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Critical Metascience ! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>

Sabina Leonelli&#8217;s new book &#8211; &#8220;Philosophy of Open Science&#8221; &#8211; will be published later this year. However, there is an open access preprint available on the PhilSci Archive here. In...

Sabina Leonelli’s “Philosophy of Open Science”

<p>I really enjoyed <a href="http://mel-andrews.com">Mel Andrews</a>&#8217; recent essay: "<a href="https://www.thephilosopher1923.org/post/philosophy-in-the-trenches-and-laboratory-benches-of-science">Philosophy in the Trenches and Laboratory Benches of Science</a>" and its main point that &#8220;every laboratory needs a philosopher.&#8221; This point is made in the context of a concern about &#8220;the industrialisation of science&#8221;:</p><blockquote><p>Not all of the shifts of recent history have constituted improvements to science, in either its truth-seeking capacity or its social responsibility. I allude to the industrialisation of science. To the corporatisation, the commodification, and the militarisation of science&#8230;.</p><p>The so-called replication crisis has come about, in large part, because generations of behavioural scientists have been instructed in assembly-line procedures rather than modes of questioning and critical analysis.</p></blockquote><p>This concern about the industrialisation of science reminded me of something that statistician Ronald Fisher wrote many years ago when he argued that the Neyman-Pearson approach to hypothesis testing is more suitable for &#8220;acceptance procedures&#8221; used during quality control on industrial production lines than for scientific research:</p><blockquote><p>I am casting no contempt on acceptance procedures, and I am thankful, whenever I travel by air, that the high level of precision can really be achieved by such means. But the logical differences between such an operation and the work of scientific discovery by physical or biological experimentation seem to me so wide that the analogy between them is not helpful, and the identification of the two sorts of operation is decidedly misleading (<a href="https://doi.org/10.1111/j.2517-6161.1955.tb00180.x">Fisher, 1955, pp. 69&#8211;70</a>).</p></blockquote><p>As I&#8217;ve discussed elsewhere, the production line analogy assumes an inappropriate amount of knowledge and certainty on the part of scientists (<a href="https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/repeated-sampling">Rubin, 2020</a>).  Unlike quality controllers on a factory production line, scientists don&#8217;t know the relevant and irrelevant features of their populations and testing conditions. Factors that they theorize to be essential for demonstrating an effect may turn out to be unimportant, and factors that they assume to be unimportant may turn out to be essential! Indeed, one of the key differences between scientific research and quality control is that scientists can <em>learn </em>what&#8217;s important and unimportant from their replication failures, whereas quality controllers merely consign their &#8220;bad batches&#8221; to the garbage. Hence, like Andrews, I see a clear link between the replication crisis and the industrialisation of science (e.g., <a href="https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/what-is-a-type-i-error">Rubin, 2021, p. 5829</a>).</p><div class="captioned-image-container"><figure><a class="image-link is-viewable-img image2" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg" width="1023" height="757" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg","fullscreen":null,"imageSize":null,"height":757,"width":1023,"resizeWidth":null,"bytes":216910,"alt":null,"title":null,"type":"image/jpeg","href":null,"belowTheFold":false,"topImage":true,"internalRedirect":null}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F482eb29e-586d-4b27-8e53-f99411b5254f_1023x757.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="#FFFFFF" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></a><figcaption class="image-caption"><a href="https://www.flickr.com/photos/jurvetson/5201796697">Steve Jurvetson</a></figcaption></figure></div><p>Andrews also describes the industrialisation of science as a:</p><blockquote><p>broad-sweeping prohibition against &#8220;theorising&#8221; &#8211; and by this it is meant any and all forms of sustained critical thought about or questioning of the scientific work</p></blockquote><p>Again, this description rings true for me. In particular, it reminds me of <a href="https://doi.org/10.1177/135910530000500306">Kerry Chamberlain&#8217;s (2000)</a> description of &#8220;methodolatory,&#8221; which consists of:</p><ol><li><p>An overemphasis on locating the &#8216;correct&#8217; or &#8216;proper&#8217; methods</p></li><li><p>A focus on description at the expense of interpretation</p></li><li><p>The avoidance of theory, and</p></li><li><p>Avoidance of the critical</p></li></ol><p>As Chamberlain explained &#8220;methodolatry&#8230;works to prevent us looking at the assumptions behind our research.&#8221; Andrews&#8217; proposal of a philosopher-in-the-lab should provide an important safeguard against methodolatory:</p><blockquote><p>Every laboratory needs to have someone who understands broadly and intimately what is going on and who is empowered to ask questions, to stand back from the laboratory bench or the telescope or the command line and ask: <em>What are we really trying to determine here, and to what end? Will it be of benefit or detriment to mankind? Are the assumptions and idealisations that we have made along the way warranted? Where do our driving theories, models, and representations of the phenomena under study stand in relation to rival or complementary conceptualisations? How are we best to interpret and contextualise the results of our modelling or experimental work? Are our instruments of measurement and analysis, our conceptual repertoire, up to the task at hand?</em></p></blockquote><p>Andrews concludes that:</p><blockquote><p>Every laboratory needs a philosopher. The philosopher&#8217;s role, however, is not only to philosophise on behalf of scientists, but to re-educate scientists in how to philosophise for themselves. </p></blockquote><p>This seems like a good idea. In recent years, statisticians and methodologists have made excellent contributions to re-educating scientists in the wake of the replication crisis. Philosophers should be key members of this educational team! </p><div><hr></div><h3>The Article</h3><p>Andrews, M. (2023). "Philosophy in the trenches and laboratory benches of science": An essay by Mel Andrews. <em>The Philosopher, 111</em>(1). <a href="https://www.thephilosopher1923.org/post/philosophy-in-the-trenches-and-laboratory-benches-of-science">https://www.thephilosopher1923.org/post/philosophy-in-the-trenches-and-laboratory-benches-of-science</a> </p><div><hr></div><div class="subscription-widget-wrap" data-attrs="{"url":"https://markrubin.substack.com/subscribe?","text":"Subscribe"}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Critical Metascience! Subscribe for free to receive new posts.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p></p>

Critical Metascience

Open Science and Overwork

Opening Up Open Science to Epistemic Pluralism

Questionable Metascience Practices

Sabina Leonelli’s “Philosophy of Open Science”

The Industrialisation of Science