What is BFMI (Bayesian Fraction of Missing Information)?

I recently gave a talk to the excellent research team at Signal Media. And got asked the question I didn’t want to be asked. It was what is the BFMI in PyMC3?

The way I largely think of it is, in a practical level – which is – if the BFMI metric is below the threshold of 0.2 your model is bad. However I understand that this isn’t a deep understanding of it, and could be irritating to some.

Let me use the illustration from a paper by Michael Betancourt.

From a practical point of view if you had a model that returned (b) in the picture above, you’d know something is wrong. And you’d know the model is misspecified. This doesn’t tell you how to fix your model, but model building and tweaking in Probabilistic Programming is tricky, and involves intuition/ exploration.

What does it mean technically?

Hamiltonian Monte Carlo proceeds in two phases — the algorithm first simulates a Hamiltonian trajectory that rapidly explores a slice of the target parameter space before resampling the auxiliary momenta to allow the next trajectory to explore another slice of the target parameter space. Unfortunately, the jumps between these slices induced by the momenta resamplings can be short, which often leads to slow exploration.

This slow exploration is a problem, because ideally you want your Hamiltonian Monte Carlo to explore the space as quickly as possible. Otherwise in practical time – i.e. the time you have to run a model – your model won’t give good estimates.

If that doesn’t satisfy you, you can look here for more details.