Growth Experimentation Velocity

"When systems are still in flux, you don't want to overinvest in tooling that will become outdated immediately when your data schema gets updated or some other piece of infrastructure changes. However, it is essential to have a way to rapidly iterate, and that means quick access to experiment results data. So if you need to in the early days, build something simple and scrappy at first, and over time evolve it to support the team's needs."

"When it comes to product nudges and defaults, it’s worth testing a few iterations to see if you can find a win. Staying focused on our North Star metric (total dollars raised) also allowed us to recognize when to accept decreases in other submetrics."

"We invested heavily in our testing infrastructure. For example, we built a sandbox environment designed to allow us to rapidly experiment (e.g. a modularized website, Optimizely, Mixpanel, etc.). To do this, we had stakeholder buy-in from the start, and super-high autonomy."

"Testing/conversion lift studies (CLS): regularly run by marketers to validate what performance would look like if you switched a channel off, or scaled spend up or down."

"Overall, we found that Dropbox teams that adopted DRICE were able to move their key metric by twice as much as teams that stuck to a simpler prioritization process."

"One thing you’ll notice is that although there are many ways to look at it, in the end it always comes down to your best guess at ROI — the ratio of (1) effort to launch, and (2) expected impact."

"Every project you ship is an opportunity to learn about what works or doesn’t. Re-evaluate ROI after you get feedback. If you see standout results around a particular hypothesis, double down with additional related feature work."

"To get the best long-term gains, you should always have a sense of urgency. The quicker you launch winning experiments, the quicker those changes impact your growth. Not only that, but these improvements compound!"

"When I left in 2012, we were running more than 100 concurrent experiments any given time, and incredibly fast paced. Anybody in the product team could still have and idea in the morning and have it running live on customers in the afternoon."