The R2R debate, part 3: my response for the motion

The Researcher to Reader (R2R) conference at the start of this week featured a debate on the proposition “The venue of its publication tells us nothing useful about the quality of a paper”. I’ve already posted Toby Green’s opening statement for the proposition and Pippa Smart’s opening statement against it.

Now here is my (shorter) response in favour of the motion, which is supposed to be a response specifically to Pippa’s opening sttement against. As with Toby’s piece, I mistimed mine and ran into my (rather niggardly) three-minute limit, so I didn’t quite get to the end. But here’s the whole thing.

Here I am giving a talk on the subject “Should science always be open” back at ESOF 2014. (I don’t have any photos of me at the R2R debate, so this is the closest thing I could find.)

Like the Brexit debate, this is one where it’s going to be difficult to shift people’s opinions. Most of you will have come here already entrenched on one side or other of this issue. Unlike the Brexit debate, I hope this is one where evidence will be effective.

And that, really, is the issue. All of our intuition tells us, as our colleagues have argued, that high-prestige journals carry intrinsically better papers, or at least more highly cited ones — but the actual data tells us that this is not true: that papers in these journals are no more statistically powerful, and more prone to be inflated or even fraudulent. In the last few days, news has broken of a “paper mill” that has successfully seen more than 400 fake papers pass peer-review at reputable mainstream publishers despite having absolutely no underlying data. Evidently the venue of its publication tells us nothing useful about the quality of a paper.

It is nevertheless true that many scientists, especially early career researchers, spend an inordinate proportion of their time and effort desperately trying to get their work into Science and Nature, slicing and dicing substantial projects into the sparsely illustrated extended-abstract format that these journals demand, in the belief that this will help their careers. Worse, it is also true that they are often correct: publications in these venues do help careers. But that is not because of any inherent quality in the papers published there, which in many cases are of lower quality than they would have been in a different journal. Witness the many two-page descriptions of new dinosaurs that merit hundred-page monographic treatments — which they would have got in less flashy but more serious journals like PLOS ONE.

If we are scientists, or indeed humanities scholars, then we have to respect evidence ahead of our preconceptions. And once you start looking for actual data about the quality of papers in different venues, you find that there is a lot of it — and more emerging all the time. Only two days ago I heard of a new preprint by Carneiro at el. It defines an “overall reporting score”, which it describes as “an objective dimension of quality that is readily measurable [as] completeness of reporting”. When they plotted this score against the impact factor of journals they found no correlation.

We don’t expect this kind of result, so we are in danger of writing it off — just as Brexiteers write off stories about economic damage and companies moving out of Britain as “project fear”. The challenge for us is to do what Daily Mail readers perhaps can’t: to rise above our preconceptions, and to view the evidence about our publishing regimen with the same rigour and objectivity that we view the evidence in our own specialist fields.

Different journals certainly do have useful roles: as Toby explained in his opening statement, they can guide us to articles that are relevant to our subject area, pertain to our geographical area, or relate to the work of a society of interest. What they can’t guide us to is intrinsically better papers.

In The Adventure of the Copper Beeches, Arthur Conan Doyle tells us that Sherlock Holmes cries out “Data! Data! Data! I can’t make bricks without clay.” And yet in our attempts to understand the scholarly publishing system that we all interact with so extensively, we all too easily ignore the clay that is readily to hand. We can, and must do better.

And what does the data say? It tells us clearly, consistently and unambiguously that the venue of its publication tells us nothing useful about the quality of a paper.

References

• Carneiro, Clarissa F. D., Victor G. S. Queiroz, Thiago C. Moulin, Carlos A. M. Carvalho, Clarissa B. Haas, Danielle Rayêe, David E. Henshall, Evandro A. De-Souza, Felippe Espinelli, Flávia Z. Boos, Gerson D. Guercio, Igor R. Costa, Karina L. Hajdu, Martin Modrák, Pedro B. Tan, Steven J. Burgess, Sylvia F. S. Guerra, Vanessa T. Bortoluzzi, Olavo B. Amaral. Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. bioRxiv 581892. doi:10.1101/581892