Ronny Kohavi
Ronny Kohavi, PhD, is a consultant, teacher, and leading expert on the art and science of A/B testing. Previously, Ronny was Vice President and Technical Fellow at Airbnb, Technical Fellow and corporate VP at Microsoft (where he led the Experimentation Platform team), and Director of Data Mining and Personalization at Amazon.
Discovery & Research Skills
A robust Overall Evaluation Criterion (OEC) protects against short-term 'theatrics' by balancing primary success metrics with user experience guardrails.
"It's very easy to increase revenue by doing theatrics. Displaying more ads is a trivial way to raise revenue, but it hurts the user experience. And we've done the experiments to show that. In this cas..."
Normalizing high failure rates is essential for preventing team discouragement and maintaining the high testing volume required to find meaningful gains in mature products.
"At Bing, which is a much more optimized domain after we've been optimizing it for a while, the failure rate was around 85%. So it's harder to improve something that you've been optimizing for a while...."
Proactively documenting and revisiting past experiment wins prevents valuable product lessons from being lost to organizational churn and allows for the reintroduction of successful patterns.
"And it was sort of done, it was very beneficial, and then it was semi forgotten, which is one of the things you learned about institutional memories. When you have winners, make sure to address them a..."
Twyman's Law—the principle that any result looking too good to be true usually is—demands rigorous bug-hunting before any win is celebrated.
"We can talk later about Wyman's law, but that was the first reaction, which is, 'This is too good to be true. Let's find a bug.' And we did. And we looked for several times, and we replicated the expe..."