Special Coverage

Doing Post-hoc Explanation Right (168)

Chris Donkin and Aba Szollosi (University of New South Wales)

Summary by Taylor Curley, Digital Content Associate Editor

In Science, Missing the Target Is Okay

Have you heard the joke about the Texas Sharpshooter? A Texan fires a couple of shots at the side of a barn. He then walks up to the wall, paints a target around his bullet holes, and declares himself a crack shot.

This parable is often used to illustrate the dangers of post-hoc reasoning and to champion measures that prevent hasty changes to theories, such as a priori specification of hypotheses and preregistration of experimental studies. Because of this, many researchers have drawn the same conclusion: Post-hoc exploration is bad.

Chris Donkin and Aba Szollosi argue that such rigid thinking is unnecessary in empirical research and that post-hoc explanations are vital to the scientific enterprise. Instead of taking care to predict all possible outcomes in advance, researchers should instead focus on specifying theories that are hard-to-vary, or ones that cannot be easily changed after data is collected, but are flexible enough to explain deviations from the expected results.

Consider a situation in which your friend claims to be excellent at free-throws in basketball. In this example, a “bad” theory would be akin to aiming for a basket the size of an empty swimming pool: The target is so broad that it will capture virtually any shot, regardless of skill. In contrast, a “good,” hard-to-vary theory would be like shooting free-throws towards a normal-sized basket. What is crucial about this distinction is that your friend may not always make the basket despite being an expert shooter. However, the target is narrow enough that your friend can sufficiently explain why some shots were missed, such as a small lapse in technique or a gust of wind if the court is outside.

In short, Donkin and Szollosi’s talk reminds us that post-hoc explanations are not only valid but are also useful tools in science: Just make sure that your target well-defined.