The R2R debate, part 2: opening statement against the motion

Yesterday I told you all about the Researcher to Reader (R2R) conference and its debate on the proposition “The venue of its publication tells us nothing useful about the quality of a paper”. I posted the opening statement for the proposition, which was co-written by Toby Green and me.

Now here is the opening statement against the proposition, presented by Pippa Smart of Learned Publishing, having been co-written by her and Niall Boyce of The Lancet Psychiatry.

(I’m sure it goes without saying that there is much in here that I disagree with. But I will let Pippa speak for herself and Niall without interruption for now, and discuss her argument in a later post.)

The debate in progress. I couldn’t find a photo of Pippa giving her opening statement, so here instead is her team-mate Niall giving his closing statement.

The proposal is that all articles or papers, from any venue, must be evaluated on their own merit and the venue of publication gives me no indicator of quality. We disagree with this assertion. To start our argument we’d like to ask, what is quality? Good quality research provides evidence that is robust, ethical, stands up to scrutiny and adheres to accepted principles of professionalism, transparency, accountability and auditability. These features not only apply to the underlying research, but also to the presentation. In addition, quality includes an element of relevance and timeliness which will make an article useful or not. And finally, quality is about standards and consistency – for example requiring authors to assert that they are all authors according to the ICMJE guidelines.

And once we agree what constitutes quality, the next question is what quality assurance do the different venues place on their content? There is a lot of content out there. Currently there are 110,479,348 DOIs registered, and the 2018 STM report states that article growth is in the region of 5% per annum with over three million articles published each year. And of course, articles can be published anywhere. In addition to journal articles, they can appear on preprint servers, on personal blogs, and social networking sites. Each different venue places its own quality standards on what they publish. Authors usually only place their “good stuff” on their personal sites, reputable journals only include items that have passed their quality assurance standards including peer review. Preprint archives only include materials that pass their criteria for inclusion.

Currently there are about 23,000 articles on bioRxiv, of which approximately a third will not be published (according to Kent Anderson’s research). This may be due to quality problems, or perhaps the authors never sought publication. So they may or may not be “quality” to me – I’d have to read every one to check. Of the two thirds that are published, they are likely to have been revised after peer review, changing the original article that exists on bioRxiv (perhaps extra experiments or reanalysis), so again, I would have to read and compare every version on bioRxiv and in the final journal to check its usefulness and quality.

A reputable journal promises me that what it publishes is of some value to the community that it serves by applying a level of independent validation. We therefore argue that the venue does provide important information about the quality of what they publish, and in particular that the journal model imposes some order on the chaos of available information. Journal selectivity answers the most basic question: “Is this worth bothering with?”

What would I have to do if I believed that the venue of publication tells me nothing useful about their publications? I could use my own judgement to check the quality of everything that has been published, but there are two problems with this: (1) I don’t have time to read every article, and (2) surely it is better to have the judgement of several people (reviewers and editors) rather than simply relying on my own bias and ability to mis-read an article.

What do journals do to make us trust their quality assurance?

1. Peer review – The use of independent experts may be flawed but it still provides a safety net that is able to discover problems. High impact journals find it somewhat easier to obtain reviews from reputable scientists. A friend of mine who works in biomedical research says that she expects to spend about two hours per article reviewing — unless it is for Nature in which case she would spend longe, about 4–5 hours on each article, and do more checking. Assuming she is not the only reviewer to take this position, it follows that Nature articles come under a higher level of pre-publication scrutiny than some other journals.

2. Editorial judgement – Editors select for the vision and mission of their journal, providing a measure of relevance and quality for their readers. For example, at Learned Publishing we are interested in articles about peer review research. But we are not interested in articles which simply describe what peer review is: this is too simplistic for our audience and would be viewed as a poor quality article. In another journal it might be useful to their community and be viewed as a high quality article. At the Lancet, in-house editors check accepted articles — checking their data and removing inflated claims of importance — adding an extra level of quality assurance for their community.

3. Corrections – Good journals correct the scholarly record with errata and retractions. And high impact journals have higher rates of retraction caused by greater visibility and scrutiny, which can be assumed to result in a “cleaner” list of publications than in journals which receive less attention — therefore making their overall content more trustworthy because it is regularly evaluated and corrected.

And quality becomes a virtuous circle. High impact journals attract more authors keen to publish in them, which allows for more selectivity — choosing only the best, most relevant and most impactful science, rather than having to accept poorer quality (smaller studies for example) to fill the issues.

So we believe that journals do provide order out of the information tsunami, and a stamp of quality assurance for their own communities. Editorial judgement attempts to find the sweet spot: both topical, and good quality research which is then moderated so that minor findings are not made to appear revolutionary. The combination of peer review and editorial judgement work together to filter content, to select only articles that are useful to their community, and to moderate excessive claims. We don’t assume that all journals get it right all the time. But some sort of quality control is surely better than none. The psychiatrist Winnicott came up with the idea of the “good enough” mother. We propose that there is a “good enough” editorial process that means readers can use these editorially-approved articles to make clinical, professional or research decisions. Of course, not every journal delivers the same level of quality assurance. Therefore there are journals I trust more than others to publish good quality – the venue of the publication informs me so that I can make a judgement about the likelihood of usefulness.

In summary, we believe that it is wrong to say that the venue tells us nothing useful about the quality of research. Unfiltered venues tell us that there is no guarantee of quality. Filtered venues tell us that there is some guarantee of reasonable quality. Filtered venues that I trust (because they have a good reputation in my community) tell me that the quality of their content is likely to match my expectations for validity, ethical standards, topicality, integrity, relevance and usefulness.