Brief thoughts on qualitative and quantitative projects

Back in the first post about our recent paper on bifurcated cervical ribs in apatosaurines, I noted:

I’m fond of this one because it’s pleasingly low-tech and traditional. We looked at some fossils, noticed some interesting features, thought about what they mean, wrote it up, illustrated it with specimen photos and diagrams, and called it done. There is certainly a time and place for phylogenetic analysis, geometric morphometrics, and all the other numerical methods that are increasingly common in vertebrate palaeontology, but I genuinely think it’s important that this kind of work doesn’t squeeze out the more foundational process of looking at, and thinking about, fossils.

This has been much on my mind of late, especially as the majority of talks at SVPCA 2023 and many new papers involve numerical methods. Sometimes I feel that Matt and I are in danger of being left behind by a new wave of palaeontology, and it’s definitely true that we could usefully apply (say) geometric morphometrics to our specimens if we had the time to learn how it’s done.

And yet, and yet …

Today I came across a Mastodon thread summarising a preprint (Ploner and Stafford 2021), “How analysis strategy affects analysis results”. Stafford’s summary says, in part:

A host of […] projects have confirmed the worrying conclusion that you can have defensible analyses which produce differing results. But Sebastian and I wanted to pursue further the issue of exactly how wide the spread of results is.

Sebastian computationally generated 1000s of different possible analysis models — permuting possible covariates and interactions — to get a size of the space of possible results. The question was this: how do human teams fill space? Does expertise in analysis mean outcomes cover a restricted, perhaps tiny, zone of the possible outcomes?

The multiverse of computationally generated analyses covered a smaller range than the spread discovered by human teams! Whatever human analysis teams were doing — choosing different possible statistical frameworks, model forms, data recoding, outlier exclusion, etc — it produced more widely varying output than randomly combining covariates in a single model form (mixed model logistic regression, since you asked)

And the punchline:

This result suggests […] that there is a hidden universe of data analysis choices which can both be a) legitimate and b) poorly recorded or recognised by researchers

The important part of this, to me, is Stafford’s in-passing point that the 29 teams whose results differed so widely all made legitimate and defensible data-analysis choices. They all did good work. But they all did different good work with the same dataset, and the outcome was that they all got different results.

When I showed this thread to Matt, his comment was:

That is interesting. And a bit worrying, since a lot of the “big science” to come in future decades will be big analyses of big datasets.

I’m glad to be poking around weird anatomy instead.

I think there is wisdom in that. I’m nervous about the idea that if we did (for example) apply geometric morphometrics to a set of cervical ribs, we might get significantly different results depending on what landmarks we chose, or on other factors.

Whereas the kind of largely descriptive work we did in the recent paper has a different quality. What we wrote has no computer-generated veneer of objectivity: it’s just what we saw and what we thought about it. Our interpretations could be wrong, but that’s fine: they’re written down so they’re refutable. Heck, even our descriptions could be wrong — we might have misinterpreted structures. But that’s OK, too: other people can look at the fossils, reach their own conclusions, and argue their case about why what we wrote is wrong.

But you can’t really argue against the results of a finite element analysis or what have you. All you can do is run another finite element analysis, get different results from the team that did the first one, and say “huh, the computer spat out a different result this this”.

I would find that unsatisfying.

Again, please note: I am not saying that numerical method are without value! I’m not even necessarly saying they are less valuable than we assume (though I do think we should treat the outputs of any given numerical analysis with a bit more scepticism). I’m just saying I’m glad I don’t have to do much of that kind of work.

References

• Ploner, Sebastian, and Tom Stafford. How analysis strategy affects analysis results: assessing results space and structure of Silberzahn et el. (2018) through model specification. PsyArXiv, 8 Dec. 2023. doi: 10.31234/osf.io/b2hm7
• Silberzahn, R., E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, Š. Bahník, F. Bai, C. Bannard, E. Bonnier, R. Carlsson, F. Cheung, G. Christensen, R. Clay, M. A. Craig, A. Dalla Rosa, L. Dam, M. H. Evans, I. Flores Cervantes, N. Fong, M. Gamez-Djokic, A. Glenz, S. Gordon-McKeon, T. J. Heaton, K. Hederos, M. Heene, A. J. Hofelich Mohr, F. Högden, K. Hui, M. Johannesson, J. Kalodimos, E. Kaszubowski, D. M. Kennedy, R. Lei, T. A. Lindsay, S. Liverani, C. R. Madan, D. Molden, E. Molleman, R. D. Morey, L. B. Mulder, B. R. Nijstad, N. G. Pope, B. Pope, J. M. Prenoveau, F. Rink, E. Robusto, H. Roderique, A. Sandberg, E. Schlüter, F. D. Schönbrodt, M. F. Sherman, S. A. Sommer, K. Sotak, S. Spain, C. Spörlein, T. Stafford, L. Stefanutti, S. Tauber, J. Ullrich, M. Vianello, E.-J. Wagenmakers, M. Witkowiak, S. Yoon, and B. A. Nosek. 2018. Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science 1(3):337–356. doi: 10.1177/2515245917747646