Day 325 of the Daily · Growth Strategy

The Overall Evaluation Criterion

5 min read · 4 principles · apply it by lunch

The test won on the number you chose. Something you didn’t choose paid for it.

The question

Could you write down, before this week’s result lands, the one number your current test wins or loses on, the rate at which you’d trade that number against the things the change might damage, and the two outcomes that would stop it shipping even on a win? If those three are still unwritten when the numbers arrive, the criterion gets picked by whoever most wants the change to have worked.

The idea

In August 2012 six Microsoft researchers — Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker and Ya Xu — took a paper to the KDD conference in Beijing, subtitled Five Puzzling Outcomes Explained. The first puzzle is a bug. A fault in a Bing experiment served users visibly poor search results, and two of the company’s headline numbers rose: distinct queries per user by over ten per cent, revenue per user by over thirty. Frustrated people search again, and click ads that suddenly look relevant by comparison. Judged on either number, the right call was to ship the damage. The instrument that stops you is the Overall Evaluation Criterion, which Kohavi set out at book length eight years later with Diane Tang and Ya Xu in Trustworthy Online Controlled Experiments. It’s one quantitative measure of what a test is for, settled before the test runs: the metrics the change could trade against each other, combined into a single number, sitting beside a short list of things it isn’t allowed to break. Day 23’s North Star belongs to the company and moves rarely. This one belongs to the test, and you write it the week you write the hypothesis.

In the Daily · 4 principles and a template

The Overall Evaluation Criterion is Day 325 of 365.

In the Daily, each of its four principles is explained in full, with a one-page template to apply it the same day. A new framework arrives every morning at 7am for a year, across all fifteen courses of the FutureMBA curriculum.

$29 a month · cancel any time · 7-day money-back guarantee

More from Growth Strategy

The next four in the course.

All 24 Growth Strategy frameworks
Day 334 · Growth Strategy

The Controlled Experiment

When your numbers moved last quarter — the conversion rate that lifted, the churn that eased — what precisely were you comparing them against, and could you rule out the season, a competitor’s outage, the campaign that launched the same week and the fact that everyone knew you were watching? Every one of those is still sitting inside the figure you reported, and there is no arithmetic that takes them back out.

See the framework →
Day 343 · Growth Strategy

The Minimum Detectable Effect

For the test you’re most impatient to launch this quarter, do you know how many people would have to pass through each variant before a lift the size you’re honestly hoping for could register as anything other than noise? If you don’t, the test will still finish, it will still produce a chart, and you will still read a decision into it.

See the framework →
Day 352 · Growth Strategy

The Local Maximum

Line up the last dozen experiments you’ve run, write the size of each result beside it in date order, and look at where the line is going. If the wins are getting smaller and the flat results more frequent, you’re not running out of ideas — you’re standing near the top of the hill you happen to be on, and nothing currently on the roadmap is capable of finding you a taller one.

See the framework →
Day 361 · Growth Strategy

The Growth Operating Review

Without opening a dashboard, say what your growth flows did last month, whether your newest cohort’s retention curve is flattening higher or lower than the one before it, and what the last thousand dollars into your best channel bought. You can’t — and almost nobody can, because those numbers live in five tools owned by four people. That isn’t a memory problem. It’s the page you’ve never built.

See the framework →