Categories
Uncategorized

How to build an AI agent in a regulated industry, with the CPO of Hims & Hers

Check out the conversation on Apple, Spotify and YouTube.

Why regulated industries are taking to AI first (1:26)

Aakash: Dheerja Kaur is the CPO at Hims & Hers. Before that, she was CPO at theSkimm. She was the first engineer-PM at Grantland, and she’s one of the few consumer leaders who has shipped in finance, media and healthcare. Dheerja, welcome to the podcast.

Dheerja: Thank you. I’m excited to be here.

Aakash: Those three categories you’ve shipped in seem to be adopting AI pretty quickly. What’s your take on AI adoption in these regulated industries?

Dheerja: First, I’d just like to say my joke about my career is that I’ve worked on the vegetables, because these are things people don’t naturally like to engage with. You’d probably rather be on TikTok than in your finance app or your health app. So it’s a hard consumer problem. How do you build a delightful, engaging consumer experience in a category people sometimes have an inherently negative relationship with? I don’t have the perfect secret sauce, but it’s been a common theme across my career.

In terms of the AI piece, I do think finance and healthcare have a lot of overlap. These are systems that are traditionally super archaic and very inaccessible. Think about the markets. When Robinhood first launched, it was the first time it didn’t cost $10 to place a trade. Every single trade used to cost $10 before Robinhood. Commission-free trading was a massive democratization. I probably don’t need to tell you how bad the healthcare system is, how archaic it is and how hard it is to navigate.

So what’s happened with AI and LLMs, no surprise to anybody, is that this technology has suddenly let you break down these massive barriers. It enables what really rich people get in these industries, like financial advisors or concierge doctors, at scale. You get the quality of that experience at scale, in a way that’s super accessible.

Everybody has their own experience with AI and health especially. But I think health is the number one topic at OpenAI and Anthropic. It’s the number one use case for AI because it is so broken. And suddenly you finally have a technology that can actually connect the dots across a very archaic and broken system.

Aakash: Why is it, though, that it’s showing up in these regulated places before the fun stuff?

Dheerja: I will say finance and healthcare are a little bit different from that perspective. In finance, I don’t think we’ve yet seen the killer AI product. Everybody’s dabbling in it. But in health, it’s this acute problem that everybody feels every single day. So suddenly you have this product, and I’m sure we’re all doing this, where you can talk to it about how you’re feeling every day, your symptoms and the different components of what you’re experiencing across your health. And it is truly, and sadly, the only thing you can interact with that can actually help you across that journey.

Our healthcare system is extremely siloed. You have doctors in specialties who don’t think beyond their specialty. You have systems where you have to go to different offices, urgent care, specialists. There isn’t a single entity, a single body, looking at you holistically. And finally you have that. So I think it’s the acuteness of the problem. I think that is the number one problem in America today.

Aakash: The same reason it’s regulated is the same reason people are turning to AI.

Dheerja: Yeah, I think it’s regulation. I also think in healthcare it’s the incentive system. You’ve got insurance. You’ve got the entire system crafted around incentivizing efficiency and siloing. Again, LLMs can completely break that model.

Why Hims & Hers is going AI-native (5:07)

Aakash: Hims & Hers has gone through a pretty big transition, from point solutions for very specific healthcare needs, where maybe I want to solve one particular problem, to more of an AI-native full solution. Why that shift? Why you guys? What’s the moat around that? Why have you taken this direction?

Dheerja: For sure. When I first talked to Hims about joining, I didn’t fully understand what the company was going after. Then I had this magic moment that made it so obvious to me: if anybody is going to actually solve this problem in healthcare, it’s this company. And that’s not just a pitch. I really do believe that.

The way I’d think about it is that we believe there are three components to solving the proactive, longitudinal healthcare problem for the average consumer.

The first is diagnostics. We need to get a picture of your full body. That’s across blood testing and biomarkers, wearables data, your medical history and even more advanced testing, like genetic sequencing. There should be this living, breathing picture of your health, and unfortunately it’s really hard to get that today. Pre-AI, I think everybody had Google Drive folders with their different documentation, and fragmented data everywhere: your wearables, your history in different places. So being able to create a single living, breathing home for all your health data is really powerful. For us, the big investment has actually been in verticalizing lab testing. We’re able to ship you an at-home blood testing device, have you send it back to our own lab processing facilities and own the full data loop there, so we can get this ongoing picture of your health. That’s the first.

The second piece is the actual medications. How do we actually improve your health? This is where the company has put the most investment historically, and we’re fully verticalized there as well. We offer treatments across all these different conditions, like mental health, weight loss, hair loss, sexual health and hormonal therapy. And we have the ability to personalize those treatments. We can say, “Based on your blood test, here’s the exact dosage of cholesterol medication we’re going to prescribe for you.” Then, as your levels improve, we can change that dosage over time. Or we can combine what you need: cholesterol medication, supplements, weight loss drugs. These are all things we can provide as a full-stack solution, and be extremely personalized. I think of it almost like precision medicine.

That full loop is incredibly hard to build. Having the health data and the verticalization of blood testing, in addition to the verticalization of treatments and being able to offer the thing that helps people improve, takes years and years of capital investment to build. Hims has that today.

The third piece is what you’re getting at, which is the actual care and doctor experience. We have over a thousand providers on our platform today who prescribe and diagnose for these different treatments and medications. We’re building the AI layer on top of that. Can we take the best of both worlds? You have what LLMs can do in terms of connecting the dots, providing proactive support and looking at that closed loop of data to ask, “What does that mean for what your treatment should be?” Then you have real humans in the loop for the actual diagnosis and prescribing, and that human support. Again, we have all of those components.

So the consumer experience we’re building is this. Imagine a world where you’re a Hims or Hers member. On an ongoing basis, you have lab testing and diagnostics. You have your hyper-personalized treatment plan. And then you have what feels like a concierge doctor in your pocket, with AI and real doctors in the loop. I think we’re one of the only companies that actually has the capabilities to build all of that, and a lot of it we’ve already built today, which is super exciting.

Inside the new Hers app (11:12)

Aakash: Walk us through it. What does the provider experience look like? What does the patient experience look like? How does that come together in the app?

Dheerja: One piece of context is that we’re very early in that new consumer experience journey. A couple of months ago, we launched a greenfield new app, and I can talk about some of that too, which is super fun. How often do you get to build a greenfield, zero-to-one app at scale? We launched a whole new app, and we started with Hers and weight loss, since that’s actually the most complicated journey we see today. Side effects, plateauing. It’s a tough journey people are on, and it’s just the very beginning. So we started with this single use case, and obviously we’re expanding from there into that full picture I described.

The experience we built is a super simple consumer product. It’s a two-tab app. Your home screen is what we call a group chat with your agent and your doctors, where it feels like a single-threaded conversation. It is a single-threaded conversation, where you’re getting proactive support from the agent and responses to your questions. Then, any time it requires a real doctor’s guidance, they’re just part of that conversation. There’s no switching. Today, people are sometimes in ChatGPT or Claude, and then they have to copy and paste that and bring it to their doctor. This is just one seamless consumer experience, and it’s pretty magical.

The other part of it is, like I mentioned, this living, breathing picture of your health. For Hers weight loss customers, we actually ship you a free Bluetooth scale as part of your membership, and that auto-syncs to your AI companion and your app. So you step on the scale, and if it notices you might be plateauing, it reaches out to you: “Maybe we should consider talking to your doctor and changing your dose. Let’s bring your doctor into the conversation.” Or, “How are you feeling? How are those side effects doing?” It’s a very high-touch, proactive experience that connects your data to this agent that feels like a companion. And again, the doctor is right there in the conversation, so you’re not bouncing between all these different places.

It’s very magical. I’m using that word a lot because I sit there every single day and read all the conversations people are having. I can’t express enough how amazing it is to have agents and doctors in one conversation. No passing you between people, no context switching for you as a customer. It’s pretty awesome. So that’s the consumer experience.

What the AI does for doctors (13:39)

Dheerja: Then, on the provider side. For those of you not familiar with healthcare, there’s a term called EMR, an industry term for how doctors look at a patient chart. When you go to the doctor, you probably see they have your chart up, and they’re looking at your history and all of that. We have that tool for our doctors. It’s a big part of what we build.

Aakash: So, electronic medical records.

Dheerja: Exactly. We’ve always had that. But now we’ve integrated AI into that product for doctors. There’s a super streamlined summary of the patient’s medical history, their current treatment plan and anything to know about side effects they’ve experienced. Then it also tees up what’s happened in the conversation so far. So in that exact moment, the doctor is able to say, “Okay, got it. Understood. Let me respond in a way that makes it clear I actually have the right context.”

Then the best part is that when that goes into the conversation, and hopefully you’ll see this in the demo, the agent takes the handoff back from there. It asks, “Do you have any follow-up questions?” Things like that. It’s really cool, and I’m really proud of it. And again, like I said, it’s just the very beginning.

How doctors, PMs and engineers build it together (14:44)

Aakash: So obviously there was a team of PM, engineer, designer and, I guess, this other key team member: doctor.

Dheerja: Doctors, yes.

Aakash: Building this agentic product, I imagine you had a lot of nervousness, or I would as a product leader, around making sure it’s built correctly. Walk us through how that collaboration works and how you actually build this type of product.

Dheerja: Absolutely. There are so many layers to this. First, it’s important to understand that this company has been around for years and has 10 years of clinical data and outcomes to train off. That’s massive, because we’re at scale. We’re serving millions of customers. So you’re not starting from scratch, which is massive in the LLM space, obviously.

Then a big part of the way we build products in general, but especially this product, is that we have a clinical team. We have medical leads in each of our specialties, like weight loss and hormonal therapy, and they’re experts in their field. They’re incredible. It’s actually really fun for PMs and engineers, because you get to sit side by side with them. You take what they call clinical protocols, which are documented guidelines for each category and its treatments. You get really detailed on this stuff. Think of it as a doctor’s training codified into writing. And then you get to turn those into code.

Mo, our CTO, likes to say this a lot: we treat clinical protocols as code. They get checked in. If they’re ever changed, that gets checked in as well. So even our clinical team gets trained on how to write, how to be an engineer, kind of. Obviously a lot of the PM and engineering job is to translate that in the right ways. But they’re sitting side by side every day, every week, as they build.

Then, once this thing is in dogfooding, there’s a daily clinical standup to triage what’s going on and what we’re seeing. There’s a whole set of protocols around that. They’re truly embedded on the team and part of the cross-functional team.

How evals work on a live health agent (16:48)

Aakash: The topic everybody keeps talking about is evals.

Dheerja: Yes.

Aakash: It’s pretty much the new PM skill that everybody needs to master. How do evals work in this context?

Dheerja: Very common in this space, we follow eval-driven development, and I think that’s definitely a key part of the PM job. We also have an entire evals team on the applied AI team. We go through the classic loop of eval-driven development, where you’re testing and you’re releasing. We also have an LLM judge that’s live in production, scoring an answer before it even gets sent to a customer. So we have a couple of different components there.

The rubric that made the agent feel human (17:27)

Dheerja: I would say it’s also more than evals, to be honest. One of my favorite examples from our AI PM team: when we first started playing around internally with v0 of this AI companion, we all felt it was really cold and very clinical, because it was built off just these clinical protocols. We hadn’t yet built the actual human layer and the consumer experience we wanted to create.

So one of our PMs, as just one example of many, put together a personalization rubric. How do we actually define the type of personal conversation we want to have with our customers? That’s across nine different dimensions. It’s anything from “Does it know you?” Is it aware of your context? Does it know your treatment plan, your journey, your goals, the emotional context you’re in? The second is mapping that to the clinical protocols and guidelines, and synthesizing the information in the right way. And the third is voice and tone. What type of warmth are we looking for? That actually varies between our two brands, even.

Those are the types of things I see our PMs doing, not just evals. To me, that’s such a key part of how you become a consumer PM in AI.

Aakash: The two brands being Hims and Hers.

Dheerja: Yeah, exactly.

Aakash: Two different experiences.

Dheerja: Yeah. And even between the different treatment journeys, you might have different approaches and different tones. That can get a little bit complicated, and there are trade-offs to that. But the reality is, somebody on a weight loss journey might look different from somebody on a peptides product, who’s different from somebody on hormonal therapy. So how do you work that in and codify it?

Who writes the evals (19:03)

Aakash: So it’s a lot more than evals, but let’s go deep on evals for a second. Where do they start? Are doctors writing the first evals?

Dheerja: Doctors are not writing evals. Doctors are codifying the clinical protocols, like I mentioned, and those get checked in. Then, between our PM team and our evals team, those are turned into evals.

Aakash: Okay. So what does the evals team focus on, and how do the PMs collaborate with them? Let’s make it more concrete. Say, a hundred responses: these are good responses, the tone here is good.

Dheerja: Exactly. Basically, you’re writing your evals based off the clinical protocols, for example. How do I define what’s accurate from a clinical perspective? Then you’re layering on these additional dimensions, like the personalization rubric. We have a rubric, so let’s create an eval off of that. Does that make sense?

Then you’re evaluating live, in real time, as you run these things in the wild, and iterating on your evals based on that. Your offline evals are really important when you’re developing the first version of the product. But after that, you’re mainly focused on what’s live and how people are experiencing it.

Aakash: Okay, got it. Does the evals team still manage that once it goes live? What’s the PM collaboration model with them?

Dheerja: It’s a small team. When I say evals team, we have one applied AI team across PM and engineering, and there’s an evals component of that team. I would say this feels very much like a startup. We have a small group of people in a daily standup looking at conversations, but also looking at the LLM judges, the evals and what’s being triggered in production, and triaging them every single day. Then we have a lot of internal dogfooding as well.

Early results: 3x engagement, 80% resolved by AI (20:54)

Aakash: What can you share about the early success of this product?

Dheerja: It’s really exciting. Like I said, we’re only a couple of months in, rolled out as an experiment. It’s not fully at scale yet, but we’re already seeing 3x the customer engagement, which is kind of insane. And we’re seeing the AI companion resolve 80% of conversations without needing a human in the loop, which is just massive. To have that on day one is kind of crazy.

And that’s even before we’ve really started to scratch the surface on being proactive, not just reactive. What we’ve started to do since launch is have the companion proactively reach out to you. Like I said, if it notices your weight is plateauing, or you hit a milestone, it says, “Let’s celebrate that. Let’s check in on how you’re doing.” That’s really where your engagement loop comes into play, and we haven’t even scratched the surface on that. So it’s super exciting.

Specs in Git and a team OS (21:43)

Aakash: Talk to me about the team behind this and how they work. How has that changed? Are you embracing the AI-pilled way to work? And what are the components that have been most successful for you?

Dheerja: For sure. This team especially, these are all AI folks at their core, across engineering and PM. Doctors are now getting AI-pilled too, because they’re all operating in the new-ish way of working. I say new-ish because I feel like we’re all there. It’s some of the stuff I think you’ve talked about on your podcast.

If you talk to the team, they’re not using classic project management tools or classic sprint planning processes so much as operating extremely lean and extremely fast. They have a Git repo where PRDs, if you even want to call them that anymore, specs, are committed into Git. Sometimes you’re just auto-generating PRs based off of that. You’re operating in a super code-based environment, where you’re getting from spec to code in hours, if not minutes sometimes. And obviously you have all the checks and balances on top of that. So it’s fun.

Aakash: Specs in GitHub, huh? What does that look like?

Dheerja: Oh, yeah. Actually, most of our teams are doing that now.

Aakash: That’s crazy. So is planning run out of GitHub too?

Dheerja: Planning has not been solved yet, let’s put it that way. The quarterly and yearly planning process around business goals, what our roadmap bets are and all of that, is still a little more traditional. What we’re seeing is that on-the-ground product execution is getting a lot more like what you talked about on a past podcast, a team OS type of setup.

Aakash: What have been the benefits of a team OS for you guys?

Dheerja: One is a shared context layer. Every team operates a little bit differently, but they’re all pulling from the same data and the same information. This is table stakes stuff, but a lot of the nitty-gritty manual tasks, like reporting on data, experiment decisions, and flagging and monitoring gaps and what’s happening in production, all happen in one single place everybody can pull from.

What we’re really seeing with the AI team is that, yes, there’s this shared context layer, but then there’s a workflow side to it. Like I mentioned, you’re getting specs checked into Git. Engineers are pulling those down. Those can get auto-generated into code at some stages, and sometimes designs. Every team has a different flavor of that workflow, but I think that’s the dream state.

Wild West mode for new AI tools (24:14)

Aakash: Very cool. So what’s your take as the leader? Are you pushing your teams to embrace all the latest AI workflows as soon as they come out? How are you properly piloting and figuring out which of these things you’re going to take on? Because it feels like there’s a new one every single day. Now they’re talking about Instinct. Are we all going to embrace Instinct all of a sudden?

Dheerja: I think we’re probably doing what most companies at this stage are doing, which is that you can’t be prescriptive. If you say, “Okay, everybody’s now going to use this tool, and this is exactly how we’re going to do it,” then you have to go through change management every single month, if not sooner. You just don’t do that.

So we’ve been pretty explicit with the teams across product, engineering, design and even the business teams that we’re going to operate in Wild West mode, and it’s going to be fun. Here are some tips and tricks and tools. We set up a Git repo for a basic team OS setup, which we actually took inspiration from your podcast on. There’s also a PM OS version of that, for PRD templates and things like that. So people can get up and running if they don’t have a place to start from.

Aakash: Nice.

Dheerja: But then make it your own. And then have tons and tons of forums for people to share what they’re doing. It’s been awesome. It’s been very organic, and it’s felt like everybody’s a builder again and trying stuff. I think you have to just embrace the Wild West of that.

We’ve also been hesitant to go out and adopt new enterprise-level tools, for that reason. Then you have to get into change management and onboarding, and by the time you’ve done that, is there a new and better tool out there? So we’ve let teams use the basics. Obviously, everybody has Claude, et cetera. Build your own tools and see how far you can get with that. For example, our TPM team built an entire in-house roadmap tool for us that auto-pulls from Jira. So I definitely feel it’s way too early to know what the right processes and tools should be, and you’ve got to be in this Wild West and figure it out.

Give people access, then don’t be prescriptive (26:17)

Aakash: I’m hearing three different things in your strategy. Number one, you’re actually getting people access. A lot of the healthcare PMs I’m talking to, although you said it’s obvious, don’t even have access to Claude. They’re stuck on Microsoft Copilot. Maybe they have ChatGPT, but they don’t even have MCPs or API connections.

Dheerja: Totally.

Aakash: So give people access. Number two, give people scaffolding if they want a starting place.

Dheerja: Totally. No cold start.

Aakash: On what we think could be a good thing. For you, it sounds like PM OS and Team OS are that scaffolding. And the third thing, which maybe is the most important, is don’t be overly prescriptive. Let your best people discover what the new stuff is and then bring it to the rest of the team.

Dheerja: 100%. Totally.

Aakash: It sounds simple because we just made it three ingredients, but I would guess 90% of teams aren’t doing that.

Dheerja: I always wonder that, if I’m totally honest. As a product leader in this day and age, I worry every night: are we behind? Everybody has that. The anxiety around being AI-pilled enough is real. I think that’s true if you talk to folks in the industry. And then of course there’s the anxiety everybody has: I have agents that can do so many things. Am I doing enough?

So I’m not sure, actually. At least in tech companies, and to your point, industry by industry this looks different. Most of what I’m seeing is that everybody’s just building again and figuring it out. Everybody’s coming at it with this mentality of, “We don’t know what this job is going to look like. We should completely first-principles this, play around and have fun.” I’ve said this before: I’m having the time of my life. This is the most fun I’ve had in my entire career since I was an engineer, and I think that’s super fun.

How the CPO job has changed (28:09)

Aakash: I imagine there’s a spectrum, for sure. At the CPO level, what’s changed? How has your life changed?

Dheerja: Ooh, that’s a good question. I was talking to my husband about this, because in some ways the anxiety is there because a lot of my job hasn’t changed. A lot of the tactical pieces of it have changed. I think a lot of people have chief-of-staff agents teeing things up, even teeing up the prep for this podcast, and running down meeting prep and doc creation and all that fun stuff. I also created a Dheerja AI agent to try to codify my product judgment. So I’m playing around with this stuff.

But what I’m finding is that those are still more efficiency gains for my job. A lot of my job is running a team, setting strategy and direction, reviewing work, giving feedback and figuring out where the consumer product is going.

Aakash: Not to mention hiring a good team.

Dheerja: Yeah. So it’s true that I’m trying to see how much of that judgment and role I can keep codifying, to really get the step-change benefits. But I’ve kind of paused on a lot of the my-job aspect of it. The mentality I should embrace is that most of my job should be about enabling this on the teams. That’s where you get the most leverage as a business. So it’s less about AI-pilling the CPO job, and more that my role is shifting to: how do I unlock the benefits of this technology for the teams? And getting into the details on that. Trying to code stuff myself, trying to play around with different workflows myself. That’s been the fun part.

Product manager or product builder? (29:45)

Aakash: Coding stuff yourself, and you mentioned being a builder. What’s your take on this? LinkedIn actually renamed their APM program to Associate Product Builder. We recently had Srini Raghavan on, the CPO at Freshworks. He thinks the PM role might be replaced by the product builder role in the future. Do you think that’s likely? Are you embracing the product builder role?

Dheerja: It’s so funny. This reminds me of when Airbnb, I forget how long ago it was, said, “We don’t believe in the PM job anymore.” There are a lot of these hot-take narratives, and I ignore a lot of them, because I think it really is just a constant renaming exercise, reinventing the fact that you still need this fundamental job. What you call it and how you title it just doesn’t matter. I love that. Product builder, great.

I do think product manager, if you really boil it down, is kind of a crappy title, because it implies that all we do is ringlead, and that we don’t actually have judgment, or we don’t build and aren’t hands-on. But transparently, even pre-AI, the best PMs were builders at heart. So I think you can get lost in the spicy hot takes and narratives on LinkedIn. I love it. Great, we can all call ourselves product builders now. I don’t think that actually changes the fundamentals of the job and what you look for.

What I think you are seeing, and I’m sure you’ve seen this a lot when you talk to folks, is that a couple of things are happening with the PM role, or PD or PB role, now. One is the folks who spike on the fundamentals. I’m talking judgment, instinct, taste, all the things we say are important in AI, plus drive, hustle, ownership, build, get shit done. Those things spike even more than they used to.

Aakash: Yeah.

Dheerja: But those were the things most companies looking for great product people already looked for. Horsepower. Then there are the people who spike more on just the tactics of the job but don’t necessarily spike on that first bucket. I think that’s the real tension. The reality is that in the ZIRP era, with hypergrowth in tech, the PM job exploded, so you have quite the spectrum of skills. It’s true that there are often different flavors of PMs. I think those who spike in the fundamentals will just be the most valued people at a company, though I’m a little biased, obviously. And for those spiking more on the tactics, sure, you can reinvent yourself and all of that. But the tactics are getting automated away.

The fundamentals she hires for (32:17)

Aakash: You briefly covered the fundamentals. But if you had to define them in clear buckets, the three to four skills you should be thinking about, what would those be?

Dheerja: For sure. And again, I will say this has not changed. I’ve looked for this over the last decade of product hiring. It’s easy to anchor, like I said, more on the outputs and the tactics, and not do enough to interview for the fundamentals. But the fundamentals of what makes a great PM have not changed.

The first is what I just said: core product intuition, judgment, decision-making and taste. It depends on what type of role you’re talking about. For an AI PM versus a pure consumer PM versus a platform PM versus a growth PM, how judgment and decision-making manifest looks a little different. But the reality is that the highest-leverage person for a company is somebody who can make the right calls. They use the reps they’ve had and the intuition they’ve learned, or just the raw intuition they have, to make the right calls. That’s it. That is the number one thing.

The second, especially when you apply this to more junior folks, is high slope, high horsepower. You just know. You can interview for this, and there are ways to interview for it. It’s somebody who’s a sponge, super high intellect, high horsepower, high slope. They maybe don’t have as much of the first, sometimes, because they just haven’t been around the block enough. They don’t have the reps. But the second they see something, they pattern-match, pick it up and apply it the next time. That, I will pick over experience any day, over deficiencies in the other areas.

These are all soft because, like we’re saying, the tactics of the job are dramatically changing. So I don’t interview for tactics as much anymore. Yes, you understand they can do the job, but these are the things that get them to do the job well. The third is just ownership and drive. You’re just hungry.

Aakash: Being high agency, too.

Dheerja: Oh my God, so high agency. Especially when you’re in these regulated industries, to be honest. I think the best PMs at Robinhood were the ones who would say, “Okay, there’s all this ambiguity around this legal question, for example. I’m going to go read the law. I’m going to figure this out, partner with legal and just embrace that.” Those people have always made the best PMs at every company. And again, the tactics of how you execute look so different now than they did four years ago. I think really smart people who have those three things can figure that out.

Aakash: Yeah, they can adapt, especially if they’re given the right scaffolding.

The interview that picks people apart (34:45)

Aakash: So how do you find these PMs now? What does your process look like these days to hire PMs?

Dheerja: We do interview loops. We do case studies. We’ve integrated AI into all of that. I try not to repeat things I think people have heard a lot. My favorite interview is maybe old school of me. We’re going to spend an hour, and you’re going to take me through the hardest product or problem you’ve ever shipped. I need you to take me through every micro piece of it along the way. We’re going to go really deep. We’re talking, “Okay, wait, this metric moved, but this didn’t. What about this? What were the four experiments you shipped right after?”

It’s the best pressure test of whether you can take a curveball, in the judgment and decision-making along the way. There are a lot of PMs who can talk the talk in case studies and interviews. There are books on these things. So really putting them to the test, like “Explain every little decision you made along the way, how you corralled people and the gnarly problems you had to work through,” usually just picks people apart.

Now we’re also trying to figure out how you test for people who are builders, going back to what you were saying earlier, because the way you execute the job just looks different. With your classic prototypes before PRDs, are you ready to get your hands on the keyboard and try something, versus just writing docs and sending them over the fence? I think there are ways to do that, and we’re playing around with a few different ones. But again, maybe I’m just old school. I think those fundamentals are going to be true no matter what age we’re in.

Prototypes before PRDs (36:23)

Aakash: It’s interesting that you mentioned prototypes before PRDs. When we were talking about specs in GitHub earlier, I wasn’t sure if specs still played a key role. What did we used to use the PRD for? Basically, it was a gating document.

Dheerja: Yeah.

Aakash: I’m gating that legal has signed off on this, compliance, product marketing, everybody. So is there still a role for something like that?

Dheerja: It’s interesting. One thing is that the PRD used to be the beginning of the process. And for a lot of products, a picture says a thousand words. Being able to articulate not just my problem statement, but also “Here’s the thing I’m envisioning,” is also just more fun. I’m telling you, what PM would prefer to write a doc to explain it, rather than make something cool?

I think that happened pretty early on in AI. It was one of the first use cases for PMs: “Whoa, I can bring this thing to life. It’s so much easier to explain the way I’m thinking about this, and then design and I can jam on it from here.” So that’s been great.

Aakash: That has been around for a while. Just to pause, though: not everyone is sold on that, if I’m honest.

Dheerja: Interesting. Really?

Aakash: If you look at the Itamar Gilads of the world, who are very much about evidence-based product management, or even Marty Cagan, he has cautioned against this. He says if you start building the prototype, you can anchor on the solution way too soon. But you guys haven’t found that to be a problem?

Dheerja: No. I think a lot of that depends on the talent itself and the partnership between product and design, assuming you have design. If you don’t, then you’ve got to do it. Then that is it. The best designers in the world are the ones who say, “Okay, I think I understand what you’re saying, but I’m not going to take that at face value. Let me rev on it.”

Honestly, we do this among the exec team too. We’ll just exchange protos to rev. We had this Slack channel for what the app looks like in the future, and it was myself, our founder and a bunch of us literally exchanging Claude protos every day. It was just fun, and you’re revving on each other. So as long as you have people who understand this is meant to be illustrative, and who know how to say, “Okay, great, I get where you’re coming from, but let me boil that back down and come up with my perspective,” it’s magical.

Aakash: You’re not committing to the solution. You’re really using this to discuss.

Dheerja: 100%. But I do agree that anything in the wrong context, or without the right dynamics in a working relationship, could totally stagnate things. Because now suddenly you have PMs who have to be designers.

Aakash: Which isn’t necessarily the right skill.

Dheerja: Yeah. That’s the thing.

How AI shows up in the case study (39:10)

Aakash: So we were talking about hiring, and you mentioned that you have case studies and they’re integrated with AI. Can you open that up a little more? What are your case studies? How are they integrated with AI?

Dheerja: For sure. I’m thinking, should I give the case study prompt? No, because we interview a lot of people.

Aakash: No. But what’s the nature of it? Is it a homework assignment? Is it a slide?

Dheerja: Sorry, I was trying to work out whether I should share our case studies. I probably shouldn’t.

Aakash: Sometimes case studies, if you’re in big tech, are just a 45-minute hypothetical they make you work.

Dheerja: Yeah. We did this at Robinhood too. We played around with whether we should do homework, a take-home or not. And transparently, people are really busy, especially the best people. I think the more you can avoid making people do homework, the better. I just do. I also think the element of people thinking on their feet shows you a lot. I really think it does.

Sure, maybe you have to lower your bar for the complete amount of output you can get out of an hour of people on their feet, versus a few days or a week at home. But we played around with it a lot at Robinhood, and I found it was just asking a lot for people to take something home, especially when you’re trying to talk to people who have jobs already. Who’s got the time for that? And two, you don’t really know how much of it they were able to do, and how fast. So thinking on your feet is really good, especially for a case.

So we do a classic product case, honestly very similar to historical big tech interviews, tied to our industry. We start with a basic prompt, and you’ve got to take me through it from there. I think the main AI integration is “Show me, and play around with it.” We’re batting around different ways to do this, but can you actually show me how you use AI to ideate and prototype off of that?

Aakash: Okay. So people will pull up their laptop, and you’ll watch them actually use AI. Are they prototyping? How are they prompting it? Is it a quality prompt?

Dheerja: The reason I’m pausing is that the industry is evolving so quickly, I don’t know how good a signal that is anymore. Most people at this point are using Claude and other tools to prototype. So I don’t know how much signal you get about someone’s AI fluency from doing that. However, I do think a signal of a good PM, going back to our earlier conversation, is somebody who can illustrate their ideas, because suddenly the onus is on you to do that. You’re not just writing docs and sending them over a wall. But I’m open to tips and ideas. We’re figuring it out.

The PM archetype that’s fading (41:40)

Aakash: I always like to zone in on the hiring component, because I think the type of PM who succeeds in today’s world may not be the same as five years ago. There’s a certain amount of fundamentals that is really, really important. But there was a certain type of PM. I interned at Google.

Dheerja: Yeah.

Aakash: I think they didn’t even get access to talk to customers, ever. You know what I mean? Corporate came up with the future, so they didn’t even need pure strategy. What they were really good at was working within an organization.

Dheerja: Yeah.

Aakash: That, I feel, has become a little bit less of an archetype that can succeed than it could in the past.

Dheerja: I agree. That’s kind of what I was saying earlier about the spectrum. The number of PMs out there has just exploded over the last decade, maybe less. And transparently, what that’s meant is that, like I said, you have people who spike on different things, and some of those things might not be as relevant anymore.

There was definitely a long period of time where the pure executor PM, who knows how to corral cross-functional partners, had value. That’s an incredible skill. It had so much value, because you’ve got to corral. You’ve got to wrangle the people. You’ve got to herd the cats. You’ve got to drive decisions forward, all of that. I think that will still be relevant, but not at the level it has been. And like I said, there’s the judgment, the taste, the first-principles thinking, the structured thinking, all that stuff. If you don’t have that and you only bring execution, I think that’s going to be tough for folks.

What makes an AI PM different (43:23)

Aakash: You mentioned AI PMs, growth PMs, core PMs. So you still have a role that is AI PM. Why is that a different role? Why isn’t every PM just an AI PM now?

Dheerja: It’s pretty straightforward, in that we are actually building AI products. And I think anybody can become an AI PM, for what it’s worth. I don’t think you need deep experience in AI, because who actually has that? Or ML backgrounds and things like that. I think the best consumer PMs can become AI PMs, but the role does look different. The way you build an AI product, like what we just talked about and described, is different from a classic growth PM doing conversion optimization and things like that.

Aakash: Yeah.

Dheerja: So from a skill set perspective, whether you come into it with this or you learn it, you have to understand the technology at a deeper level and how to work with those engineers. And it just looks very different from what product development historically looked like. So that’s really the difference. Are you actually building an AI product, or are you a PM who uses AI, which should be everybody?

Aakash: So the PMs building AI products are PMs who spike in AI fundamentals and in teaming with AI research and engineers. They’re really good at building AI evals and teaming with the AI evals team. Is that it?

Dheerja: At least partnering. And I’m sure this is different from company to company. If you work at OpenAI or Anthropic, at a foundation model company, the level of depth required there is much higher. For us, it’s more of a balance: I need you to have great consumer fundamentals too, because you’re also crafting the experience of interacting with it. But you need to be able to work with the engineering team and get into the nitty-gritty, because otherwise you can’t execute. You’re just crafting a design, handing it to engineers, and they have to figure it out.

The hard part of that job, and of creating the product I described, is not the user experience. It’s the evals and the protocols. How do you create a personalization rubric as we experiment with more proactive outreach and the things the companion can do? How do you build a conversational guide for those? How you do that is kind of retraining the execution muscle. So you have to learn it, or have it coming in, which again is more rare. And the type of work you do day to day just looks different.

Inside the personalization rubric (45:41)

Aakash: You mentioned the personalization rubric a couple of times. I feel like it might mean more to you than to me. Can you break down what that is and what’s involved in it?

Dheerja: Yeah. I use it because I like to give concrete examples. Otherwise we all just talk in these terms. This is one example of many that our AI PM team created as we were building this companion. It’s what I was referring to earlier. We were dogfooding the very first version internally, and it was clinical only. It would respond to you very clinically. It felt like you were talking to a very impersonal doctor.

So a lot of the exec feedback in our product review was, “How do we layer on the fact that we know a lot about you?” Not just your clinical data, but also your goals, where you are in your health journey and who you are as a human. We know all these things from these conversations. How do we layer on tone and brand, the warmth and the conversational style? And then how do we layer on the synthesis of that information?

To the point of the conversation we’re having, that is a classic example of a skill the AI PM needs to figure out. How do I take that and codify it in a document that can then be turned into something that actually changes the user experience of interacting with an agent? I don’t think that document was ever a thing in classic product development. PRDs and specs are much more straightforward from that perspective. If you’re specking out a UI, you say, “This button does this, and this button does that.” But trying to codify that type of feedback is just a new muscle. That’s what I meant by the personalization rubric. It’s just one example of many.

Another is what we call conversational guides. Our companion proactively reaches out to you, or you ask it a question, like, “Hey, I’m feeling some side effects.” There’s the clinical protocol. But then there’s how I guide the conversation around that. You have to figure out, as a PM, how to codify that for the engineering team to run with. Does that make sense?

When the guide got too deterministic (47:47)

Aakash: Yeah. What does good look like there? How do you codify that?

Dheerja: We’re figuring it out, man.

Aakash: Okay.

Dheerja: To really go under the hood, it was really funny. With the conversational guide, we were just talking about this a few weeks ago. The very first versions of these got so prescriptive and deterministic. It’s so easy to lose sight of the fact that this is an LLM.

Aakash: Oh.

Dheerja: Just as an example: “Okay, if the customer asks about this, then start with this thing, and then do this.” You craft the response, and you get narrower and narrower, because you start getting anxious as a PM, or even as a designer or whatever. How do I make sure it’s really doing the thing I want it to do? I remember at some point the docs got super long. Then you’d read them, and we’d all say, “We’re kind of losing the point here.” The whole power of an LLM is that it’s flexing to the customer and learning about them.

So how do you find the balance? You’re writing that type of rubric or guide so it’s hitting the right points and not too off-kilter, without being overly deterministic and prescriptive. It’s crazy. What words do you use? What is the structure of that document? We’re just revving on that. It went from this really, really long deterministic thing to more of a set of guidelines, and we’re just playing around with it. You have to try it out and see what’s working and what’s not.

Should the AI PM title be split up? (49:16)

Aakash: So we talked about this title, AI PM. Some people hate this title.

Dheerja: Really?

Aakash: They’re really against it.

Dheerja: Why is that?

Aakash: Because they say it’s putting too many roles into one label. Actually, the head of product at Sarvam AI just had a LinkedIn post on it this morning. He said, “We’re putting way too many roles under the label AI PM.” We really need to talk about it. Maybe you have your AI evals PM. Somebody I was just working with got hired into a purely evals role. Maybe we need an AI model PM, because maybe you decide, “Oh, we should actually be custom training our models.” Then maybe you have your AI feature PM, who would probably own the personalization rubric we just covered. So what do you think about that? I know we’ve talked a lot about titles today, but people seem to really zone in on hating this one.

Dheerja: Oh, man, that’s interesting. I was always somebody who was fundamentally allergic to the over-specialization of PM titles in general. This is akin to back in the day, when it was, “Is growth PM a thing? Is platform PM a thing?”

Aakash: I grew up as a growth PM.

Dheerja: There you go. So my default mode is to wait until it hits a breaking point, where you’re seeing such different profiles of people, and some are clearly successful and some are clearly not, for that to warrant this specialization. And of course it also depends on to what end the title is important. Is it for hiring purposes? Is it for calibrating what the expectations of a role are? If it’s the latter, I think you can just get kind of lost in titles, to be honest.

So if I draw that analogy, there was a time when everybody was a PM, and people just spiked on different things. Why are we over-specializing everybody? Like we said earlier, the fundamentals are there no matter where you’re applying them. But then you get to the point where you realize that the people who are successful in growth versus platform versus zero-to-one consumer have dramatically different spikes. And I feel like they do. I totally agree that it’s very useful from a hiring perspective and from a career development perspective. I remember even at Robinhood, we had people who bounced between them, and it was a moment: “Okay, you’re now going to try consumer for the first time. Let’s make sure you’re learning these skills.” So it’s very valuable from that perspective.

I don’t know if we’re there yet with AI. It feels a little early. And fine, let it be the Wild West for now. To me, at least for now, maybe it depends on where you work. If you’re at a foundation model company, maybe that specialization need is there. I don’t think we’re seeing that at our company.

Aakash: That’s why Sarvam cares.

Dheerja: Makes sense.

Aakash: Because they’re building a foundation model, so they need the specialization. But if you’re sitting on top of the models, it’s a little bit less important.

Dheerja: Exactly.

How anyone can become an AI PM (52:05)

Aakash: You also said anyone can become an AI PM. A lot of people want to, not just for the career development and the market signaling. It seems AI PM roles pay a little better too. So how does anyone become an AI PM?

Dheerja: Okay, a couple of things. One, I think you have to be at a company where this is core to the bet and core to the consumer product, if you want to do it within a company, which is oftentimes the easiest way to learn something and transition into something. The reality is that there are going to be a lot of companies and products that are looking for their AI feature, but it’s just not core to the strategy, no matter how hard you try. So I think the most straightforward path is to work at a company similar to what I’m describing, where AI is the thesis of how that company wins, because then you’re in the thick of it. Whether you actually move into a full AI PM role at the company or not, no matter what you work on, you’re getting exposure to how to build AI products. That’s such a natural graduation, because you’ve got to get the reps in somehow.

Separately, on the flip side, I don’t think there are that many AI PMs out there, because it’s a whole new world. Everyone calls themselves an AI PM. To our point earlier, what does that really mean? Is it that you use AI, or are you building an AI product? And which part of that product? So again, it’s kind of the Wild West right now.

At least for us, and I think a lot of companies do this, you just take bets on great PMs. It’s still early enough that, maybe not anybody, but a lot of companies aren’t saying the number one skill set they need is somebody who has done this before, versus a great PM with the right instincts for the space they’re in and high horsepower, who has people to learn from. In our example, our AI PM team is a bunch of folks who haven’t necessarily worked in AI product, but they’re working with applied AI engineers who are best in class at it. So I know they’ll learn it from them.

My advice is that oftentimes the best way to learn is adjacency within the company you’re in.

Aakash: Yeah.

Dheerja: So if you’re in a company where AI is at its core, go for it. It’s awesome. We’ve given internal folks a lot of opportunities to do that within our company. And alternatively, I think people will hire you if you’re a great PM. It’s still the Wild West out there.

The upskilling path if you can’t move internally (54:32)

Aakash: A lot of people listening to that are saying, “Well, I can’t move internally at my company.” And maybe they’re ready to leave anyway, for whatever reason. Is there an upskilling path you think could be useful, whether it’s side projects or content or something like that, to get yourself prepared for that next AI PM role?

Dheerja: Yeah. What’s going through my mind right now is that there’s so much you can do to learn how to use AI effectively, like reading your newsletter and podcast and playing around with stuff on your own. But to go a layer deeper, to say, “I can call myself an AI PM, which means I actually know how to build AI products,” that’s a meaningful investment. That means maybe you’re actually trying to build a product yourself on the side.

Aakash: It’s worthwhile, you think?

Dheerja: Yeah. Right now, building is so fun that if you have the time for it, go for it. I actually know a lot of people who are taking a mini career break or a pause so they can just get in there and learn the technology. So my high-level encouragement is that, to me, it’s less about consuming content. If you can carve out time in your week, which for most PMs doesn’t exist, build hands-on and try to go deeper and deeper into it. Not just the fun stuff, like “I built a little home management assistant,” but going layers deeper and trying to unpeel that.

Aakash: You have an AI model that you’ve done something with, whether it’s fine-tuned or it’s using RAG or something like that.

Dheerja: Exactly. Which, again, is hard to do on your own in a side project, but a lot of people are doing it. And I think that could be great.

Aakash: And it’s actually just a few prompts away, or a couple of n8n workflows. It’s easier than it ever was.

The PM who built his way onto the AI team (56:24)

Dheerja: Exactly. I’ll give this story. We have a PM who was on our pharmacy and fulfillment systems. We build our own software for our pharmacies. In addition to that, he was one of our early adopters of AI and helped build the PM OS and things like that. Then we gave him an opportunity to join the AI team, because he said, “I want to,” what you’re saying, “go multiple layers deep and actually learn how to build AI products.” And hopefully it’s okay to say this, but a few weeks in, he said, “Okay, I can do this. It’s not that much of a black box.” So I do think him doing it on his own, on the side, gave him a lot more priming than he realized.

Aakash: Nice. That’s a really concrete example. So what you can do is some side project stuff.

Dheerja: Yeah.

Aakash: You can try to enable the overall PM team with some of the stuff you’re learning. That created the confidence in you.

Dheerja: Yeah.

Aakash: To say, “Okay, you can move into this new team.” Sometimes people think side projects are against your current job. But the way this person did it, it seems like he framed it the right way to you.

Dheerja: Yeah. Right now, in this day and age, learning on your own and having the intellectual curiosity to say, “I just want to learn everything I can about this new thing because it’s so cool,” is something I love leaning into, at least for me. And showcasing that, whether it means you’re moving into an AI PM job or we’re just totally elevating it. I think that’s just so necessary for the culture you want at companies these days.

I think it’s very different from the past, when side projects were truly side projects. These aren’t really side projects. One of the best things I think PMs on our team can do is have the intellectual curiosity to embrace what their jobs could look like in the future and what they could build. I know that will pay dividends for their job day to day. So we have to foster and embrace that, for sure.

The biggest mistake AI PMs make in regulated industries (58:29)

Aakash: We’ve had two major threads in this conversation. In the first half, we talked about how to build successful AI products in regulated industries like healthcare. In the second half, we talked about how you can become a successful AI PM, and whether it’s really a role. I want to end on this note: what are the mistakes AI PMs in healthcare or regulated industries make, so that if somebody’s listening to this, they can avoid making them?

Dheerja: It’s probably going to be pretty obvious. What makes a great PM in these regulated industries, even pre-AI, is really understanding, and fundamentally, intellectually embracing, the fact that you are in a regulated place. Whether it’s your doctors in healthcare, or often your compliance friends and your legal team in finance, they are builders with you. There’s just a mindset you have to have. What I observe for folks not coming from those industries is that it’s a big learning curve, because oftentimes those are ancillary teams.

Aakash: Yeah, those are just sign-off teams.

Dheerja: They’re just sign-offs. Exactly. They’re just sign-offs in a PRD. It’s so different in these spaces. You have to build a solution together. You just do. Oftentimes, half the battle of building these products is understanding the constraints.

Aakash: Yeah.

Dheerja: And then figuring out the creative solutions. What’s awesome is that the best companies in these industries have hired people in these roles who are creative solution thinkers. I’m obviously biased, but our Hims & Hers legal team is world-class. They’re so good at what they do. They know their stuff, they understand what we’re trying to do in the industry, and they know how to partner with builders to do it. That’s incredible. But you have to totally rethink your relationship with those functions compared to other industries.

So, going back to the point of your question, one is embracing that and checking your biases on how to work with those functions. And then just recognizing the magnitude of what you’re doing. If I go back to the example of what we’ve shipped, we’ve launched an agent that is the front door of the consumer experience, directly and autonomously interacting with customers. That’s awesome. And it also comes with real risk. So it’s not a one-time solutions thing. It’s an ongoing engagement. We have our legal and clinical partners in the room every day in a standup, looking at what’s going on in production and monitoring all of that. There’s just an MO for how you operate. If you don’t have it, you might have blind spots and make mistakes. That’s also my job, to make sure that doesn’t happen. But I think it’s also inherent to the PM role.

Your next step: start building (1:01:13)

Aakash: All right, guys. We just walked you through a live case study of a company with a very successful product. This is a public company. They went through the agentic transformation. They put it at the front door of the product in a regulated industry. This is what all companies are going to end up doing in the future, and you just got a taste of what the role looks like.

Dheerja just said this off camera, but their entire team is using the Team OS that we published in this newsletter and on this podcast. Go pick one of those up. Like she said, go check it out in your side time. You don’t necessarily need to wait for your company to give you the scaffolding.

You’re one of the 1% of people who go way above and beyond and listen to the end of this podcast. So, like she said, now shift from the content to doing. Go send a couple of prompts over to Claude Code. Here’s my instruction to you: “Claude Code, let’s build an AI product together. I want it to have RAG and fine-tuning. I’m passionate about X. Where should we start?” It can literally help you from that simple prompt.

So go learn the tools, go get the job. Go apply to Hims & Hers. Go apply to a regulated industry. As you see, they’re actually embracing AI totally, in a way that surprised me. Good luck with your career, and we’ll see you in the next episode.

Leave your thoughts