Day 139 of the Daily · New Technology & Superintelligence

The Held-Out Ten

5 min read · 4 principles · apply it by lunch

Good morning. Everyone on your team has a favourite model. Not one of them can prove it.

The question

When you last rewrote a prompt or moved to a newer model, what did you compare the new version against? If it was the two or three outputs you happened to glance at afterwards, you haven’t measured anything. You’ve changed your mind and called it a test.

The idea

Ron Kohavi ran the experimentation platform at Microsoft and, before that, data mining at Amazon; with Diane Tang of Google and Ya Xu of LinkedIn he wrote Trustworthy Online Controlled Experiments (2020), the standard text on testing changes at scale. Its most useful number is also its most deflating: of the ideas Microsoft put through controlled experiments, only about a third improved the metric they were built to improve. A third changed nothing. A third made things worse. These weren’t careless ideas. They were the surviving proposals of experienced product people who had argued for them in rooms full of other experienced people. The lesson carries straight into AI work, where you change a prompt, a model or a tool almost weekly and judge the change by whether the last few outputs felt better. The trap is that you’re both the person who chose the change and the person marking it, working from memory, on a sample of two. Without a fixed set of cases and a known-good answer for each, you can’t tell an improvement from a good mood.

In the Daily · 4 principles and a template

The Held-Out Ten is Day 139 of 365.

In the Daily, each of its four principles is explained in full, with a one-page template to apply it the same day. A new framework arrives every morning at 7am for a year, across all fifteen courses of the FutureMBA curriculum.

$29 a month · cancel any time · 7-day money-back guarantee