Day 157 of the Daily · New Technology & Superintelligence

The Bake-Off

5 min read · 4 principles · apply it by lunch

Before the renewal date. Every demo you’ve ever watched was driven by someone who already knew the answer.

The question

When you last chose a tool your team now runs on every day, what sat under the decision beyond a demo somebody else drove — and why haven’t you run it against your own work since?

The idea

In 2022 Percy Liang and the team at Stanford’s Center for Research on Foundation Models published Holistic Evaluation of Language Models, known as HELM, and its lasting contribution was less about which system won than about the shape of the question. Before it, models were reported on wildly different tasks, and some prominent ones had not one of the core HELM scenarios in common. HELM ran thirty models across forty-two scenarios and, on its sixteen core scenarios, measured seven things wherever it could: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. Efficiency sits in that list on purpose, because cost and speed are part of what a system is rather than a footnote to it. The argument transfers directly to your buying decision. No single number describes a system, and the number a vendor leads with was chosen by the vendor. What you need isn’t a better leaderboard; it’s the same battery of measures, run on scenarios that belong to you.

In the Daily · 4 principles and a template

The Bake-Off is Day 157 of 365.

In the Daily, each of its four principles is explained in full, with a one-page template to apply it the same day. A new framework arrives every morning at 7am for a year, across all fifteen courses of the FutureMBA curriculum.

$29 a month · cancel any time · 7-day money-back guarantee