AI Evaluation Strategy

"Instead of focusing on shiny demos, look for rigorous data and evaluations. I’ve seen too many PMs present flashy demos to executives only to stumble when asked about data and evaluations (including some lessons I learned the hard way!)."

"Evals (short for “evaluations”) are structured ways to measure how well an AI system performs on specific tasks, such as correctness, safety, helpfulness, or tone. They define what “good” looks like for your AI system and help you answer the question: Is this model doing what I want it to do?"

"Evals are the only way you can break down each step in the system and measure *specifically* what impact an individual change might have on a product, giving you the data and confidence to take the right next step. Prompts may make headlines, but evals quietly decide whether your product thrives or dies."

"LLM-based evals allow you to generate classification labels in an automated way that resembles human-labeled data—without needing to have users or subject-matter experts label all of your data."

"Clearly articulating what you want your judge-LLM to measure isn’t just a step in the process; it’s the difference between a mediocre AI and one that consistently delights users. Building these writing skills requires practice and attention."

"Gathering data to evaluate, writing evals, analyzing the results, and integrating feedback from evals is an iterative workflow from initial development through continuous improvement after launch."

"After years of building AI products, I’ve noticed something surprising: every PM building with generative AI obsesses over crafting better prompts and using the latest LLM, yet almost no one masters the hidden lever behind every exceptional AI product: evaluations."

"As a user, you want evals that are (1) specific, (2) battle-tested, and (3) test for specific areas of success."

"Many teams build eval dashboards that look useful but are ultimately ignored and don’t lead to better products, because the metrics these evals report are disconnected from real user problems."

"The process that tells you where to focus is referred to as “error analysis” and should result in a clean and prioritized list of your product’s most common failure modes."

"The expert’s task is to provide two things for every user interaction with your AI, grouped by session: a binary pass/fail judgment and a detailed critique."

"It is through the process of reviewing outputs and articulating what feels “wrong” that the true criteria for success emerge."

"Demo value isn’t user value. Building a cool AI demo doesn’t mean we have a product that customers love and is useful."

"Hallucination is a technical term that refers to the model’s propensity to return nonsensical or false completions depending on what’s asked of it. Basically, the model is like a very smart and overeager 6-year-old. It will try its best to give you a good answer even if it doesn’t know what it’s talking about."

"In our framework, product builders work in a continuous loop of development (CD) and calibration (CC). During development, you scope the problem, design the architecture, and set up evaluations to keep non-determinism in check."

"Once you’ve deployed, you enter the calibration loop, where you observe real behavior, figure out what broke, and make targeted improvements. With every cycle, the system earns a bit more agency."

"Since most products start cold, aim to gather at least 20 to 100 examples up front. This dataset helps you evaluate system performance and also tells you what context your assistant needs in order to perform reliably."