The R2R debate, part 5: what I actually think

I’ve written four posts about the R2R debate on the proposition “the venue of its publication tells us nothing useful about the quality of a paper”:

• part 1: opening statement in support
• part 2: opening statement against the motion
• part 3: my response for the motion
• part 4: the video!

A debate of this kind is partly intended to persuade and inform, but is primarily entertainment — and so it’s necessary to stick to the position you’ve been assigned. But I don’t mind admitting, once the votes have been counted, that the statement goes a bit further than I would go in real life.

It took me a while to figure out exactly what I did think about the proposition, and the process of the debate was helpful in getting me the point where I felt able to articulate it clearly. Here is where I landed shortly after the debate:

The venue of its publication can tell us something useful about a paper’s quality; but the quality of publication venues is not correlated with their prestige (or Impact Factor).

I’m fairly happy with this formulation: and in fact, on revisiting my speech in support of the original proposition, it’s apparent that I was really speaking in support of this modified version. I make no secret of the fact that I think some journals are objectively better than others; but that those with higher impact factors are often worse, not better.

What are the things that make a journal good? Here are a few:

• Coherent narrative order, with methods preceding results.
• All relevant information in one place, not split between a main document and a supplement.
• Explicit methods.
• Large, clear illustrations that can be downloaded at full resolution as prepared by the authors.
• All data available, including specimen photos, 3D models, etc.
• Open peer review: availability of the full history of submissions, reviews, editorial responses, rebuttal letters, etc.
• Well designed experiment capable of replication.
• Honesty (i.e. no fabicated or cherry-picked) data.
• Sample sizes big enough to show real statistical effect.
• Realistic assessment of the significance of the work.

And the more I look at such lists, the more I realise that that these quality indicators appear less often in “prestige” venues such as Science, Nature and Cell than they do in good, honest, working journals like PeerJ, Acta Palaeontologica Polonica or even our old friend the Journal of Vertebrate Paleontology. (Note: I am aware that the replication and statistical power criteria listed above generally don’t apply directly to vertebrate palaeontology papers.)

So where are we left?

I think — and I admit that I find this surprising — the upshot is this:

The venue of its publication can tell us something useful about a paper’s quality; but the quality of publication venues is inversely correlated with their prestige (or Impact Factor).

I honestly didn’t see that coming.