Day 343 of the Daily · Growth Strategy

The Minimum Detectable Effect

5 min read · 4 principles · apply it by lunch

Your last test came back inconclusive. That result was settled before the test ever ran.

The question

For the test you’re most impatient to launch this quarter, do you know how many people would have to pass through each variant before a lift the size you’re honestly hoping for could register as anything other than noise? If you don’t, the test will still finish, it will still produce a chart, and you will still read a decision into it.

The idea

Jacob Cohen spent most of his career as a professor of psychology at New York University, and in 1962 he did something nobody had bothered to do. He took the 1960 volume of the Journal of Abnormal and Social Psychology and asked of each study, one by one, whether it had ever been capable of finding the effect it went looking for. On average it hadn’t: a typical study had less than an even chance of detecting a medium-sized effect even when the effect was really there. His answer was Statistical Power Analysis for the Behavioral Sciences, published in 1969 and much expanded in a second edition in 1988, which sets out the arithmetic binding four quantities together — your sample size, the size of the effect, how confident you demand to be that a result isn’t chance, and the probability you’d catch a real effect if one existed. Fix any three and the fourth is decided for you. Run that arithmetic before the test rather than after it, and the useful output is the minimum detectable effect: the smallest true lift your traffic can tell apart from noise on the settings you’ve chosen. A flat result from a test that never had the power is not an answer to the question you asked. It’s an answer to a different one — how many people did you send.

In the Daily · 4 principles and a template

The Minimum Detectable Effect is Day 343 of 365.

In the Daily, each of its four principles is explained in full, with a one-page template to apply it the same day. A new framework arrives every morning at 7am for a year, across all fifteen courses of the FutureMBA curriculum.

$29 a month · cancel any time · 7-day money-back guarantee