Day 334 of the Daily · Growth Strategy

The Controlled Experiment

5 min read · 4 principles · apply it by lunch

Good morning. Something worked last quarter. You still can’t say what.

The question

When your numbers moved last quarter — the conversion rate that lifted, the churn that eased — what precisely were you comparing them against, and could you rule out the season, a competitor’s outage, the campaign that launched the same week and the fact that everyone knew you were watching? Every one of those is still sitting inside the figure you reported, and there is no arithmetic that takes them back out.

The idea

Ronald Fisher spent the 1920s at Rothamsted, an agricultural research station north of London, working out why decades of field trials had produced so little anyone could trust. His answer, set out in The Design of Experiments in 1935, was randomisation. Assign the treatments by chance and every difference you never thought of — soil, drainage, weather, the farmhand’s mood — spreads itself evenly across both groups instead of hiding inside your result. He illustrates the machinery in the second chapter with a lady who declares she can taste whether the milk went into the cup before the tea or after. Fisher doesn’t argue with her. He designs eight cups, four made each way, presented in an order she can’t anticipate, and settles in advance how many she would have to get right before he’d believe her. Rothamsted lore holds that she was a colleague and that the test was really run; the book claims neither, and the design is the part that matters. Ron Kohavi, Diane Tang and Ya Xu set out the online version in Trustworthy Online Controlled Experiments in 2020, and nothing essential changed: a group that gets the thing, a group that doesn’t, and chance deciding which is which. The trap is the before-and-after chart. Compare this month with last month and you’ve credited yourself with the season, the competitor’s outage and everything else the world did while you weren’t looking.

In the Daily · 4 principles and a template

The Controlled Experiment is Day 334 of 365.

In the Daily, each of its four principles is explained in full, with a one-page template to apply it the same day. A new framework arrives every morning at 7am for a year, across all fifteen courses of the FutureMBA curriculum.

$29 a month · cancel any time · 7-day money-back guarantee

More from Growth Strategy

The next four in the course.

All 24 Growth Strategy frameworks
Day 343 · Growth Strategy

The Minimum Detectable Effect

For the test you’re most impatient to launch this quarter, do you know how many people would have to pass through each variant before a lift the size you’re honestly hoping for could register as anything other than noise? If you don’t, the test will still finish, it will still produce a chart, and you will still read a decision into it.

See the framework →
Day 352 · Growth Strategy

The Local Maximum

Line up the last dozen experiments you’ve run, write the size of each result beside it in date order, and look at where the line is going. If the wins are getting smaller and the flat results more frequent, you’re not running out of ideas — you’re standing near the top of the hill you happen to be on, and nothing currently on the roadmap is capable of finding you a taller one.

See the framework →
Day 361 · Growth Strategy

The Growth Operating Review

Without opening a dashboard, say what your growth flows did last month, whether your newest cohort’s retention curve is flattening higher or lower than the one before it, and what the last thousand dollars into your best channel bought. You can’t — and almost nobody can, because those numbers live in five tools owned by four people. That isn’t a memory problem. It’s the page you’ve never built.

See the framework →
Day 008 · Growth Strategy

The Flywheel

Does your business have a flywheel — and if you doubled investment in one component, would the whole loop accelerate?

See the framework →