AI Product Strategy
Prioritize high-impact workflows and navigate non-deterministic development to build defensible AI products.
The Guide
5 key steps synthesized from 18 experts.
Identify high-friction chore wedges
Return to first principles to define your product's core problem. Look for high-volume, repetitive tasks where user effort yields a disproportionate payoff if automated. Identify specific customer superpowers before deciding on a technical implementation.
Featured guest perspectives
"I think in the AI space, we're trying to hear from customers, what do you wish Slack could do if it had these new superpowers? Let's incubate a couple teams or prototype, give them space to run and pilot and then get something to launch that's amazing. Blows people away. That's the formula that we've seen."— Noah Weiss
"So it wasn't like, 'Oh, we have this cool technology, what can we do with it?' It was like, 'Let go of what you've built. Go back to the objectives you were trying to solve and now with this technology, how can you do that objective better?'"— Tomer Cohen
"Start with a core workflow that feels like a chore where the promise-to-payoff is high if you get it right. You want to select a place where the upfront user effort (like taking the leap to try it out or customizing it) yields a big reward (like substantial time savings) and invites repeat use."— Lenny Rachitsky
Select the right optimization technique
Determine if your goal is to specialize behavior or provide dynamic context. Use RAG for applications requiring live data or internal repositories. Implement fine-tuning when the model must adopt a specific style or expert craft standards.
Featured guest perspectives
"RAG is a technique that gives models access to additional information at run-time that they weren’t trained on. It’s like giving the model an open-book test instead of having it answer from memory."— Lenny Rachitsky
Design for non-deterministic outcomes
Shift from a deterministic mindset to one of continuous calibration. Build product experiences that account for squishy outputs through post-processing filters and feedback loops. Adopt a development lifecycle that explicitly manages the tradeoff between agency and control.
Featured guest perspectives
"LLMs allow writing shitty software to be significantly cheaper, not necessarily good software, but good enough in certain contexts. And also it means that there's certain software now that isn't plain old computing that can be run cheaply. It's relatively expensive marginal cost."— Alex Komoroske
"AI systems behave differently. They introduce non-determinism on both ends: in other words, there’s unpredictability in how users engage and how the system responds."— Lenny Rachitsky
"Most people think about AI-assisted services in terms of the model quality, but model quality is just a tiny piece of the total product. It turns out that post-processing filters, contractual guarantees, data privacy, feedback loops, observable human impact, etc. are all far more important."— Lenny Rachitsky
Implement the agency ladder
Scale agentic systems safely by starting with human-in-the-loop suggestions. Monitor behavioral signals to determine when a system is reliable enough for higher autonomy. Identify blind spots by having the AI suggest actions to operators before interacting with customers.
Featured guest perspectives
"You need to be deliberately starting in places where there is minimal impact and more human control so that you have a good grip of what are the current capabilities and what can I do with them and then slowly lean into the more agency and lesser control."— Aishwarya Naresh Reganti + Kiriti Badam
"How long you stay in each version depends entirely on how much behavioral signal you’re seeing. You’re optimizing for understanding how your AI behaves under real-world noise and variation."— Lenny Rachitsky
Build a defensible product layer
As foundation models commoditize, shift focus to proprietary data ownership and the creation of superior, vertical-specific workflows. Invest in the application layer and opinionated design to bridge the gap between raw model capability and user utility.
Featured guest perspectives
"I think it's a 50/50 of both. I think the capabilities obviously have improved a ton and we've seen these get better and get measurably better. But I think the other side of it is everything to do with yeah, really the product interface and the tools and so on."— Scott Wu
"So now our hypotheses are just wrong. So at this point then, most of the value is probably not going to accrue at purely the, at least this is our belief, at the infrastructure layer. It's going to accrue somewhere else. Where is the layer that you can actually differentiate on? And we believe the application layer is a very, very deep layer to go out and differentiate on, right?"— Varun Mohan
"The data and the interfaces may become more important than the models themselves, which are becoming increasingly commoditized, available via open source, and pushed to the edge. ... The AI products I am most excited about leverage a proprietary or uniquely structured set of data—for which they have a license to use rather than scrape it—and a superior interface that transforms an antiquated workflow."— Lenny Rachitsky
Get this guide as an AI skill for Claude Code
Install This Skill
Add this skill to Claude Code, Cursor, or any AI coding assistant that supports Agent Skills.
Quick Install (Recommended)
Install this skill directly using npx:
npx skills add RefoundAI/lenny-skills --skill ai-product-strategy Or install all 76 skills:
npx skills add RefoundAI/lenny-skills Manual Installation
Download the skill
Download Skill (.zip)Add to your project
Create a folder in your project root and add the skill file:
.claude/skills/ai-product-strategy/SKILL.md Start using it
Claude will automatically detect and use the skill when relevant. You can also invoke it directly:
Help me with ai product strategy Or receive these install instructions in your inbox:
Guest Perspectives
Deep dive into what 17 podcast guests shared about ai product strategy.
Aishwarya Naresh Reganti + Kiriti Badam
"You need to be deliberately starting in places where there is minimal impact and more human control so that you have a good grip of what are the current capabilities and what can I do with them and then slowly lean into the more agency and lesser control."
- Start by having the AI suggest actions to human operators instead of interacting directly with customers.
- Identify blind spots by collecting human feedback on AI suggestions before increasing the system's decision-making power.
- Incrementally add complex tools, such as issuing refunds or raising engineering tickets, only after the core agent logic is proven.
Alex Komoroske
"LLMs allow writing shitty software to be significantly cheaper, not necessarily good software, but good enough in certain contexts. And also it means that there's certain software now that isn't plain old computing that can be run cheaply. It's relatively expensive marginal cost."
- Design product experiences that account for 'squishy' or imperfect AI outputs rather than assuming 100% accuracy.
- Avoid building business models, like advertising, that cannot clear high marginal inference costs.
- Stop applying mature execution playbooks to a technology that fundamentally changes the basis of competition.
Asha Sharma
"Because these models are so effective at this point, you want to start to tune them to certain types of outcomes. All of a sudden, these are these living organisms that just get better with the more interactions that happen. I think this is the new IP of every single company products that think and live and learn."
- Measure success by the team's "metabolism" in ingesting data and digesting rewards models.
- Focus IP development on the learning loops that allow the product to improve with every interaction.
- Tune models toward specific business outcomes like quality, price, or performance via post-training.
Chip Huyen
"If you talk to the users who understand what they want or they don't want, look into the feedback, then you can actually improve the application way, way, way more. Why do you need to keep up to date with the latest AI news?"
- Prioritize talking to users and analyzing feedback over reading the latest AI news.
- Focus on building more reliable platforms and writing better prompts.
- Optimize end-to-end workflows instead of agonizing over model choices.
Edwin Chen
"If you could choose the perfect model behavior, which model would you want? Do you want a model that says, "You're absolutely right. There are definitely 20 more ways to improve this email," and it continues for 50 more iterations or do you want a model that's optimizing for your time and productivity and just says, "No. You need to stop. Your email's great. Just send it and move on"?"
- Explicitly define the core values—such as truth or human productivity—that the model should prioritize.
- Avoid optimizing for 'dopamine' or engagement metrics that lead to low-quality, addictive AI behaviors.
- Prioritize behaviors that respect user time and focus rather than those that maximize interaction length or iterations.
Jason Droege
"These things take 6 to 12 months to get them truly robust enough where an important process can be automated. Like with any of these major tech revolutions, headlines tell one story and then on the ground, laying broadband means you need to dig up every single road in America to lay it."
- Allocate 6 to 12 months of development to move an AI process from a proof-of-concept to a robust automated workflow.
- Focus on the practical 'groundwork' of your specific implementation rather than being distracted by tech headlines.
- Shift from generalist data labeling to expert-led training to handle sophisticated, high-stakes enterprise tasks.
Logan Kilpatrick
"We're not going to launch some of these varied verticalized products. We're not going to launch an AI sales agent. That's just not what we're building towards. And companies who are and have some domain specific knowledge and they're really excited about that problem space, they can go into that and leverage our models and end up continuing to be on the cutting edge without having to do all that R&D effort themselves."
- Target specific industry verticals like legal or sales where general models are less optimized.
- Build custom UI and niche product features on top of existing general models.
- Leverage domain-specific knowledge to create value that horizontal AI providers are unlikely to pursue.
Marily Nika
"There is something called the shiny object trap, and I'm always telling people, "Hey, don't do AI for the sake of doing AI." Make sure there is a problem there. Make sure there is a pain point that needs to be solved in a smart way."
- Identify a clear user problem or high-level solution before reaching out to technical implementers.
- Evaluate if the problem can be solved through smarter features like personalization, automation, or fraud detection.
- Take a step back to evaluate whether your existing data can be leveraged to create a smarter feature.
Michael Truell
"We definitely didn't expect to be doing any of our own model development. And at this point, every magic moment in Cursor involves a custom model in some way."
- Identify the specific 'magic moments' in your product that off-the-shelf foundation models cannot consistently deliver.
- Invest in internal model development capabilities once generic models reach a performance ceiling for your use case.
- Focus custom model training on the specific interactions that define your product's unique value proposition.
Noah Weiss
"I think in the AI space, we're trying to hear from customers, what do you wish Slack could do if it had these new superpowers? Let's incubate a couple teams or prototype, give them space to run and pilot and then get something to launch that's amazing. Blows people away. That's the formula that we've seen."
- Ask customers what specific capabilities or 'superpowers' they want before deciding on an AI implementation.
- Incubate AI prototypes in standalone teams to allow them to experiment outside of core roadmap constraints.
- Run pilot programs to validate that an AI feature actually delivers an 'amazing' experience before committing to a full launch.
Sander Schulhoff
"But if there were one technique that I could recommend people, it is few-shot prompting, which is just giving the AI examples of what you want it to do. So maybe you wanted to write an email in your style, but it's probably a bit difficult to describe your writing style to an AI. So instead, you can just take a couple of your previous emails, paste them into the model, and then say, 'Hey, write me another email.'"
- Use few-shot prompting by pasting 2-3 previous examples of your work into the chat.
- Implement self-criticism by asking the model to check its own response for errors and then improve it.
- Identify the best-performing prompts through a process of trial and error and direct interaction.
"Studies have shown that using bad prompts can get you down to 0% on a problem, and good prompts can boost you up to 90%. People will always be saying, 'It's dead,' or, 'It's going to be dead with the next model version,' but then it comes out and it's not."
- Develop 'artificial social intelligence' to better understand and adapt to how different AI models communicate.
- Don't wait for models to become perfect; learn to elicit better performance through structured techniques today.
- Prioritize learning how to adapt your next prompt based on the specific nuances of an AI's response.
"My main advice there is choose a common format. So XML, great. If it's, I don't know, I don't know, question, colon, and then you input the question, then answer, colon, and you input the output, that's great too. It's a more research-y approach."
- Use XML tags (e.g., <instruction>, <context>) to clearly delineate different parts of a complex prompt.
- Standardize few-shot examples using common patterns like 'Question:' and 'Answer:'.
- Structure your data in formats the LLM is 'comfortable' with, such as Markdown or JSON, to improve parsing accuracy.
Scott Belsky
"I actually think that the role of AI going forward will be to have applications increasingly meet us where we are. To this day, we've always had to generalize onboarding experiences for the most part for everyone. And I'm really excited about the day when kind of products meet us where we are based on what type of user we are."
- Utilize AI to move away from static, 'one-size-fits-all' onboarding experiences.
- Build products that adapt to meet users exactly where they are based on their specific needs.
- Focus AI implementation on reducing the friction of learning curves for new users.
Scott Wu
"I think it's a 50/50 of both. I think the capabilities obviously have improved a ton and we've seen these get better and get measurably better. But I think the other side of it is everything to do with yeah, really the product interface and the tools and so on."
- Balance raw AI capability improvements with the development of specific product features and interfaces.
- Invest in the product layer to bridge the learning curve users face when transitioning to autonomous agents.
Shaun Clowes
"Well, you've got this synthesis machine, which is this LLM thing that's going to help you do synthesis, but if it hasn't got all that data to do synthesis on top of, it's got nothing. And so that means that LLMs can only be as good as the data they are given and how recent that data is."
- Prioritize data management and pipeline freshness over prompt engineering when building AI features.
- Focus on providing LLMs with comprehensive data to enable more accurate synthesis.
- Ensure data is recent, as LLM outputs are only as valuable as the recency of the information they process.
Sherwin Wu V2
"The field and the models themselves are just changing so, so quickly. They tend to disrupt themselves. The models will eat your scaffolding for breakfast."
- Build for the model capabilities expected in 12–18 months rather than current limitations.
- Avoid investing heavily in custom scaffolding that tries to 'fix' temporary model flaws.
- Assume that current model performance is the floor and plan for rapid, disruptive improvements.
"To enable a one person billion dollar startup, there might be a hundred other small startups building bespoke software. So I think we might actually enter into a golden age of B2B SaaS."
- Identify B2B SaaS opportunities that provide necessary infrastructure for one-person companies.
- Analyze second and third-order economic shifts to find niches created by massive headcount reduction.
- Build bespoke software designed to provide extreme leverage for small teams.
Tomer Cohen
"So it wasn't like, 'Oh, we have this cool technology, what can we do with it?' It was like, 'Let go of what you've built. Go back to the objectives you were trying to solve and now with this technology, how can you do that objective better?'"
- Review original product objectives and evaluate how AI can solve them fundamentally better from scratch.
- Direct teams to abandon current builds to avoid being constrained by the logic of old technology.
- Use AI as a matchmaking tool to optimize the connection between users and their professional needs.
Varun Mohan
"So now our hypotheses are just wrong. So at this point then, most of the value is probably not going to accrue at purely the, at least this is our belief, at the infrastructure layer. It's going to accrue somewhere else. Where is the layer that you can actually differentiate on? And we believe the application layer is a very, very deep layer to go out and differentiate on, right?"
- Identify if your product's underlying infrastructure is becoming a commodity due to industry convergence.
- Pivot resources toward the application layer where there is no ceiling on user experience and workflow innovation.
Make your product team AI-native
These skills work best as a system. Get a free 8-email course: map how your team works today, wire agents into real workflows, and make it stick. From the team that does this for a living.
Related Skills
Other Overlay: Building AI Products skills you might find useful.