Check out the conversation on Apple, Spotify, and YouTube.
Why PMs should talk to every customer (1:36)
Aakash: Engineering is getting faster with AI. Is product management really getting faster? One of the ways I would argue it is, is by helping you talk to more and more of your customers. Now, of course, you’ve been using Claude to help you synthesize transcripts. But have you been using it to drive automated, proactive research with every single one of your customers? How might you actually do so? If you really want to get to that next level of customer research with AI, today’s episode is for you. Greg, welcome to the podcast.
Greg: Thanks for having me.
What an AI research report shows you (2:16)
Aakash: Greg, you told me something pretty interesting. When you were chief product officer of Bolt, you would be happy to talk to 5 to 10 customers a month. That has really changed with AI. What is this new level of AI research report that you can build?
Greg: I think it’s now possible to actually have real back-and-forth conversations with thousands or even hundreds of thousands of customers every month, in a way that would previously require an army of researchers and an army of support reps to facilitate all of those conversations. And when you’re able to have all of those conversations at scale, you’re able to produce really interesting insights.
So this is an example for the company Ancestry.com of what one of these reports might look like if you’re able to have these scaled conversations with customers. One of the first things we always report on is the impact on the customer journey from a business metric standpoint. We found that having this type of personal conversation with customers doesn’t only capture insight. It also allows you to convert a broader set of this audience. In this case, these are people who’ve abandoned the process of purchasing a DNA kit with Ancestry.com.
Beyond those conversion insights, you’re also learning, for these people who specifically dropped off in that funnel, what their biggest concerns are. What are they bringing up most frequently? What’s actually driving them to abandon?
Then, when you layer on the conversion data about what people actually do after these conversations, whether they’ve finished their purchase or not, you can double-click a layer deeper into the relative conversion rates of these buckets. So not only how often they’re happening, but when they do happen, how do these people behave, and how does that compare against the baseline of everyone who engages in conversations?
That lets you see whether something is a hesitation or a concern that’s easily addressed with messaging, where the AI responds to the customer and specifically answers the question around shipping and delivery time. That’s something that can really just be solved by updating your messaging, maybe by talking about shipping and delivery time more proactively. Compare that with something like subscription confusion, where even when the AI has the chance to respond and address that concern, these people still aren’t converting at a high clip. That opens up a broader opportunity for you to dive into that problem and address it in a more holistic way with your product.
Where the conversations come from (5:05)
Aakash: Fascinating. These conversation counts are super high. How are these conversations generated?
Greg: These conversations happen when someone drops off in that flow. After they’ve entered their email address, let’s say, or their phone number, the AI reaches out over that channel. We’re getting a large percentage of those people to respond and actually engage in a back-and-forth conversation. That’s how we get to the scale where we have customers doing tens of thousands, even hundreds of thousands, of these conversations every month, and generating this more scaled insight into a specific audience at a specific point in the journey.
Aakash: So is this basically the evolution of in-app surveys, plus AI?
Greg: Yeah, I think that’s a good interpretation. What this allows is not just the survey from an insight perspective. You can dive a layer deeper. Say someone says they’re concerned about, in this example on the left here, an unexpected relative popping up and that being awkward. You get the chance to not just take that in as a piece of data. You get to double-click on it, ask follow-up questions, attempt to resolve those concerns and see how the customer reacts. That gives you a much richer sense of how these different things are actually affecting your customers.
Building the research agent in Claude Code (6:32)
Aakash: Awesome. So how do I build this?
Greg: For a lot of people, it’s not going to make sense to buy a dedicated product like Woven to accomplish this outreach. The awesome thing is that with existing tools, you can build something very similar using something like Claude Code. In about 30 minutes before the podcast, I put together a quick example of this. Here’s how it works.
It takes in a source of customer data. For the demo, it’s just a list of customers in a CSV: people, their drop-off points (this is for people who’ve dropped off in a trial), data about their usage and so on. That allows the AI to personalize the outreach and the conversation. You can also use any form of CDP or a data warehouse, like Snowflake or Segment, to power this.
Then it has some basic instructions on how to approach the conversation and how to format the email, in the CLAUDE.md file. This is all relatively straightforward stuff, but it’s important to make sure this still feels personal. It should still feel like an email you would draft back to the customer directly.
Then I filled in some basic stuff. This is for a made-up company called Cadence that does AI fitness training. So there’s some information about different escalation processes, different policies the company has around things the agent can or can’t do, and then the specific programs. For a lot of companies, this data already exists, so you already have pretty rich markdown files about your company that you can point the agent at. And even if you don’t, you can use Claude pretty easily. Point it at your website, and it’ll generate a bunch of this on the fly.
It uses all that information to personalize outreach to this customer audience. I’ll trigger that really quick in my Claude session. There’s a skill that just does the outreach sending, but you can also schedule this or have it trigger based on new entries in this CSV or from the database.
What it’s doing behind the scenes when it does this outreach is going through that list of customers and identifying the ones it hasn’t reached out to yet. It takes that metadata about the customer, personalizes a specific outreach that’s relevant to that customer, their situation and the goals of the outreach, and then sends it directly to the customer.
I can show a couple of examples of what that looks like. This is one I did last night, right before the podcast. It’s reaching out to someone who started using this app but hasn’t converted from their trial, asking specifically about their experience with the program and what they thought about how intense it was or what the level was. The customer responds, and the agent is able to get back within a couple of minutes. You’re able to continue that conversation and keep asking follow-ups to gather more information.
One of the things we found with Woven is that a lot of the value of this personal outreach exists when you’re able to respond quickly. We’ve tested adding delays to the response mechanism, and you see the reply rate drop precipitously. People are likely no longer in their inbox, no longer in the workflow of doing their email, and you lose them at that point. The conversion rate drops alongside that. That’s where AI can come in and let you respond really quickly and really accurately to these customers at scale, so you get the benefit of these long-running conversations where you’re extracting a lot of insight.
Why email beats in-app chat for this (12:29)
Aakash: So why over email? Why not just within the app? Wouldn’t that be a little bit smoother?
Greg: In the app also definitely makes sense, and we work with customers and recommend adding in-app chat as well. But email has a few benefits. One, this abandoned audience isn’t in your app anymore. You also need to be careful, when you’re doing things in the app, not to distract from the core goal of someone being in your app. You don’t know that they’re going to abandon yet, so you don’t want this research-oriented chat to end up hurting conversion instead of helping.
The other benefit is it’s really easy to roll out and scale. It’s a channel that all of your customers already use on a daily basis, so you’re able to be much more proactive and aggressive. Especially with these research campaigns, we get 20, 30, 40% of these people responding. When you think about how many people would interact with an AI chat that exists on a site, it’s much, much lower. So you get a broader ability to capture that audience.
What drives a 40% reply rate (13:38)
Aakash: Can we break down what’s the difference between driving a 20% response and a 40% response?
Greg: A lot of times it’s around the type of campaign you’re running. When you’re doing something that’s very research-oriented, and something that’s a little later in the funnel, meaning people have already shown some intent (in this case, they’re already using the app and have had experiences with it), they’re much more likely to respond and much more likely to get to that higher range of responses. We just did an experiment with a company in the fitness space, and they saw about 35% for this exact use case of trial abandonment.
As you go down, it’s often lower earlier in the funnel, say someone who just entered an email but didn’t actually finish an onboarding process. There’s less intent. But it also matters how well you do this initial outreach, how personal it is and how relevant it is to them. That’s going to drive whether people respond. It’s similar to cold email outreach in B2B sales. The better you make this email, the more likely it is that this person is going to be willing to take the time to respond.
How to write the research email (15:00)
Aakash: If you had to give your average PM at a startup a template for this email, what would it look like? Could it be this kind of plain-text version, so that it seems more human?
Greg: Yeah, I think that’s the right approach. Asking simple, direct questions that are really relevant to where the user is at this point in the journey lets you open the door to them responding and sharing some information, and then you can expand on that.
A mistake people make is too much content. They’re trying to ask too many questions at once, essentially trying to recreate their survey in one email. If you take it a little bit slower, you can let the customer lead you toward what they care about and what they want to share insights about. You can also often get to all of those questions through the back-and-forth of the conversation, at a much higher rate than if you try to stuff them all into one specific outreach.
The emails to send across the customer journey (16:03)
Aakash: If you had to think about the right set of emails for someone to create, say the four points along a typical journey, how would you map those out?
Greg: Take this example of a fitness app that has a trial. There’s one at the very beginning. There are going to be people who start that sign-up process, like I mentioned, enter an email address and never actually finish creating their account, finish the onboarding process or select a program. That’s going to be the widest part of your funnel. It’s going to be the lowest response rate, but also the highest volume, so I think that one makes a ton of sense. It can also inform the rest of your journey downstream of that.
The second touch point makes sense somewhere in this trial, potentially multiple times within the trial, where you’re getting people set up for success and then checking in on that success to see how they’re proceeding through the product.
And when we’re talking about onboarding specifically, the last one that makes sense is that a lot of the people in these trial situations aren’t converting. They’re not going beyond that trial, so it always makes sense to reach out to that group as well. One of the things we’ve seen make sense, and that’s really easy to do as a PM, is to offer some form of incentive just for sharing insights at that point. You don’t have a lot to lose at that point. So you say, “Hey, if you share things you think we could do better for your next trial, I’ll give you 20% off your subscription for the next few months.”
What we’ve seen that do is massively increase the reply rate. But it also increases the reply rate for the audience that’s already on the fence. You’re not going to reply to that email for that incentive if you weren’t already considering it. So you get a captive audience where you can both address their concerns, hopefully making them feel good about purchasing the app, and pair that with a promotion you were probably going to send anyway, to get them to take that next step and come back and convert through the funnel.
Aakash: So I guess that’s the onboarding stage. What about further? Once they’re engaging in the core app, once they’re retaining, once they’re lapsing, what are the other points along the lifecycle where this makes sense?
Greg: When you’re successfully engaging in the core app, that’s where the in-app experience and chat make the most sense. That’s where you should be capturing people in the moment, in that experience. They’re using this product on a frequent basis.
Where these more traditional channels can make more sense is, again, at these points of drop-off or abandonment. It could be an active user who has explored a new feature, or an upsell feature, for the first time but didn’t actually convert. I think that’s a really unique opportunity. Or it could be someone who was previously active, as you mentioned, but has decreased their usage in the last few weeks. Or it could be someone who canceled their subscription. That’s another opportunity similar to the trial, where every incremental conversion you save there is additive. But also, some of the most interesting insights you can find come from this audience of people who have decided this isn’t for them and the value wasn’t good enough.
Sending at scale without marketing ops (19:39)
Aakash: Okay, makes sense. Operationally, are these through your transactional email system? A lot of times marketing ops will own that.
Greg: For these tests, I did this with a standard Gmail account. That obviously limits the scale of the outreach you can do, based on what’s reasonable to do from Gmail. But it does let you do it completely by yourself, without using an existing system for marketing. This is all just happening from greg@cadence-demo.com. These are the outreach emails I sent earlier. So I think that’s very possible if your audience isn’t massive.
If you’re dealing with a larger audience, you’d likely need to partner with your marketing team to use an existing tool, or it’s really trivial to set up Postmark or another tool specifically for this use case. Honestly, it’s relatively common for research-oriented use cases to already be sending from a different service. Even with the legacy systems like Medallia and Qualtrics, those emails aren’t sent by the marketing team. They’re sent from those systems.
Aakash: So you could stand up your own product research email system.
Greg: Exactly. And that’s common best practice.
Aakash: Okay. I didn’t know that.
Greg: Yeah.
Turning research threads into an engaged audience (21:08)
Aakash: Very cool. So what I’m taking away from this is that email is a super powerful channel. You should set up this stack where you have the ability to email your customers, and you can make it a lot more intelligent by emailing them at the right time, via the right triggers, based on their lifecycle. Typical things like trial, didn’t sign up, lapsed and finally churned. Then on top of that, you can layer in instant AI responses in your voice to go deeper on the conversation and on the insights you obtain, and at the same time potentially address the concerns they have, to improve the conversion rate at that step of the funnel.
Greg: That’s right. For an individual PM, this type of thing can be very powerful as you’re trying to better understand the audience for a specific product or feature you own, and you also want to drive adoption of it. You want people to be using it, and you want to be able to follow up with those people. So you also create this audience of engaged people, and you can come back to these threads. Even if it’s manual, you can have Claude take a pass through your inbox and identify interesting folks. You say, “Hey, I’m launching a new feature. I want to see everyone who mentioned something related to this feature, and I want to reach out to them,” either with a follow-up question or with an invite link to jump on a call and talk through this feature that was developed based on their feedback.
The prompts that set it up (22:37)
Aakash: So we started with the output report. We showed you what the outcomes look like as email. You cooked it up in just 30 minutes. Can you at least get us started? Show us how you would use Claude Code, and what prompts a typical PM might use to get the scaffolding for this setup.
Greg: Yeah, it’s pretty straightforward. I mainly just gave Claude the context of trying to set up a specific outreach program from my inbox using the Gmail MCP. It’d be similar if you were going to use, let’s say, the Postmark MCP. And I pointed it at the source of the customer data. That’s one of the main things you need to figure out: where are you going to get this trigger data from?
Then I gave it instructions: “Hey, this is the context of the conversations I want to start. I want to be reaching out to people who are 10 days into their trial.” Then I let it essentially run with that context. The first pass needed some tweaking. There were things like the threading being broken on the replies, so they didn’t actually get threaded back into the email, and some of the language was inappropriate. There’s some optimization you can do just by having it run through tests like the ones I’m showing here. But even with that base instruction, you end up with something that’s working in, like I said, 30 or 40 minutes, and that you can start testing with real customers.
Adding an eval loop, live (24:15)
Aakash: There are two things I think are really interesting here. The first is that as soon as we’re going to talk to the customer, we probably need to think about what our online evals are grounded in our help center, that it’s helping drive objections down the way our sales team wants, that it’s maintaining our corporate brand voice. And then there’s this component of automation: how do we create this as a system that can quickly respond? Can you walk us through how you might build those?
Greg: For sure. For this demo, we already have this classification process. But one of the things we can showcase live is the next step, which would be adding evals as a separate step in this process that runs before the actual reply gets automatically sent to the customer. If I go over to Claude Code, I’ll zoom in a little bit here.
You can essentially just ask Claude and explain what you’re trying to output. If you’ve already put a little bit of work into this, it should be able to give you a pretty good approximation. So we can say: “We now want to add an eval step after classification for responses that are queued to be auto-sent. Focus that eval step on groundedness in the documentation. If the eval fails, rerun the generation process and save the new generation as a draft for review.”
This lets you do a couple of things. One, it sets up the process for evals, and you can then build additional evals beyond just groundedness on top of it. Two, it starts building this loop that I think is really important: when an eval fails, what happens? The eval should not just stop the whole process. Ideally, you’re using the information from that eval failure to have the agent take another pass, so that even in these cases of eval failures, you can still get to automation you feel confident in.
Aakash: This is something I haven’t seen somebody build before. So basically you’re building the evals live into the product, and here we’re focused on groundedness. What Claude is doing is building this system of judging: what is the groundedness, what is the true data source, what is the LLM judge. And then, like you said, it’s creating this loop. If the LLM judge views it as non-grounded, update to the grounded version and then actually send it.
Greg: That’s right. What we’ve seen on our end is that for the vast majority of cases where these evals are failing, you can pretty quickly get to a positive response that passes the evals and the guardrails. One thing that’s important as you’re building this is that you don’t want to be overfitting, where it’s just taking this exact eval and following it to a T. Ideally, you use the eval failure as input to another full run of your process for generating the response. That way it’s still going through that full logic loop of understanding how to respond, taking the eval just as input, versus “here are the exact instructions.”
Aakash: This is fascinating. I can’t believe I haven’t heard about this before. Basically, you’re building quality into your product so there’s continuity, so it doesn’t just stop. And at the same time, you’re probably improving your performance on your online evals. If you had a groundedness metric and you launched this loop, you’d probably see a bump in that groundedness metric, because it’s getting the feedback live.
Greg: Yeah, exactly. And that’s one of the benefits of email we talked about earlier. There’s not an expectation of instantaneous responses. It’s actually a little bit weird if you’re responding within seconds. By having a little more time, you can run these loops in a way you can’t if it were a chat-based experience, because there it would just be someone sitting there waiting three minutes for the chat to respond. That’s what allows this process to exist and still deliver a good customer experience, because there’s no need to respond within 300 milliseconds.
Checking what Claude built (29:27)
Aakash: What’s pretty fascinating is that Claude had no clarifying questions. So there’s probably some risk that it goes off in some direction we disagree with. As it’s finishing up its response here, what are we going to look for?
Greg: You want to understand specifically what changes it made and how that’s affecting your process. So look at where it’s inserting updates into this process of handling the replies, how that’s showing up in the summary it’s writing here, and make sure the summary is aligned with what you actually intended the output to be.
So: the emails are only going to run in auto-send mode. That makes sense. If it’s not on auto-send, we’re already going to draft immediately. The groundedness eval will check generated replies’ claims against the underlying documentation. I think that’s also accurate. And on failure, regenerate once with the specific issue corrected, and then save to a Gmail draft, which is what I instructed it to do. Per our conversation, the next stage of this is probably updating it to retry multiple times and get to that approved response you can auto-generate.
And that’s the power of AI, too. It’s doing things I didn’t even ask for. As part of this workflow, I have Slack notifications, like this one from the email I sent earlier, that get sent when an inbound message comes in and gets processed. It automatically added this failure emoji for the case where the email fails, which I didn’t even have to explain would be valuable.
What Woven adds on top (31:23)
Aakash: Okay, fascinating. So we just showed you how you could potentially be engaging with almost all of your lapsed customers and downloading information from them. We showed you how you can vibe code this as a system built in Claude Code, with proper evals, so you have confidence when you’re putting it in front of your customers. Not everyone’s going to want to build this themselves. So what’s the experience like if they choose Woven?
Greg: A lot of this complexity is managed for the customers who work with us, so the process looks largely the same. It’s just intended to be, one, at much greater scale, and two, a bit more conversion-oriented, like we talked about with the research-oriented campaign versus the conversion one.
Going back to that Ancestry.com example, this is again someone who abandoned their DNA kit. These outreaches get sent at scale based on integrations with your marketing automation platform or CDP, so we handle that data pass-through. Customers respond, and then the AI gets back to them. As part of setting up one of these engagement points, we generate a bunch of simulated examples, which let you give feedback to improve the responses directly. That’s how, without a bunch of back-and-forth with Claude and without having to build all of this yourself, you can quickly get to confidence. You can also see and visualize the changes the agent is making based on your feedback.
So here it created a guardrail. It created a playbook. It updated the agent context. You can review all these things and gain confidence, and also see the output and how it’s been improved. On top of that, we talked about evals and guardrails. We’ve invested a lot in defining really strong global guardrails that handle things like groundedness and the formatting of the email. In addition, as part of this training process, we set up custom guardrails based on the feedback you share and on what matters most for ensuring quality. Those run on every response and, like we discussed, get rerun until we get to a passing response.
That’s one of the things that makes this powerful. We have customers who have hundreds of these engagement points live with us right now, and it takes them 5 to 10 minutes to set a new one up once they’ve gone through this process a couple of times. We talked about all the different points in the journey. Start with something straightforward, like this cart recovery use case, and then you can start running these conversational campaigns basically everywhere you have questions about customers or want to influence customer behavior.
When to talk to customers yourself (34:37)
Aakash: When is this not the right choice? When should a PM be emailing by hand, looking up customers and making sure they talk to the customer, instead of the AI talking to the customer?
Greg: I think it depends on scale, and also on where you are in the product-market fit journey. In cases of lower scale, there’s a lot of value in having those more direct conversations. There’s also a lot of value in conversations that aren’t text-based, where you’re actually meeting with someone, building a relationship and really understanding them at a deeper level.
That’s also true when you’re in the earlier stages of finding product-market fit. You can’t fully automate that process of discovery. The AI agent is never going to have the same amount of context you have as the PM of this early-stage product. So this is for a little later in that product-market fit journey, where you already have a sense that you’re creating value and you just want to maximize that value.
The full AI research agenda for a PM (35:46)
Aakash: If you broaden it out, say you’re still at Bolt for a second. You now have this new capability. They’ve built out this system. What’s the broader research agenda advice you’d give a new PM on the team? What do all of the AI-enabled research activities they should be engaging in look like, not just this one?
Greg: Like you said, this is one piece of a broader AI research strategy. This makes sense for a constant drumbeat of research, but there are opportunities to scale the other types of research as well, particularly customer interviews. There are tools like Listen Labs and other types of products that let you automate these more voice-based, back-and-forth research conversations.
Again, I wouldn’t necessarily recommend those for an earlier-stage company. But by the time I was in the later stages of my career at Bolt, those were really powerful opportunities. We had a lot of B2B customers, but also millions and millions of shoppers on our network from a B2C standpoint. So you have an opportunity to use those automated research interviews to dive a layer deeper. You can potentially use these email conversations as an entry point to find the users who have the most interesting insights to share, and then use that next level, an AI-facilitated voice back-and-forth, to go deep with those people and get a much richer set of feedback and context from them.
Aakash: Maybe you can react to this and see if it’s about right, because what I don’t want is for people to see this episode and all of a sudden purely pursue AI-enabled research. Is it accurate that a typical PM should still be regularly chatting with customers, on Zoom calls, over text message, over Slack? I’d say multiple times a week if possible, and by multiple I basically mean more than two times a week. At least you’re having a Slack chat, or you’re texting, or you’re hopping on a call, or of course the best thing, you’re literally watching them use your product in their context. I still think a PM should probably be doing that once or twice a week.
But in addition to that, they should be setting up these AI-enabled research programs. Like you said, they should be setting up synthetic conversations with an AI avatar. They should be setting up email-based chats at key points of the journey, like we demonstrated today. They should probably also be setting up some sort of platform that’s analyzing all of their sales calls and all of their social messages. So they probably have at least these three AI-enabled activities, plus the regular discovery, which is maybe part of a continuous discovery habit loop. So there are basically four key customer research activities a PM should be pursuing. Is that right?
Greg: Yeah, I think that’s right. What these more automated workflows can enable is making those true moments of interaction and relationship building much more powerful, because you’ll be coming into those conversations much more educated on your user, on what the foundation is, and on where there are potentially interesting things to poke at and dive deeper on. So they don’t replace those real moments of human-to-human interaction where you can really dive deep and understand someone. They just make them more powerful. You waste less time in those conversations covering things you should probably already understand.
That’s one of the other things we think is powerful about the more steady-drumbeat process. When we’re generating these research reports (I showed the deck-based version, but we’re also generating more report-based versions), we send those over Slack. That gives the research a chance to really live where people are working, and they get that steady drumbeat of what customers are saying and what they’re actually doing after they share this type of feedback. That can then inform the more in-depth research you do on top of this.
How AI changes what PMs need to do (40:21)
Aakash: That is just a master class in what first-class discovery and research looks like. You had some really hot takes on product management in general, so I wanted to shift there.
Greg: Sounds good.
Aakash: How has AI changed what PMs need to be able to do?
Greg: I’ve been reflecting on this recently. One of the interesting things I felt at Uber was that the ratio of PMs to engineers was often way too high. You had teams of one PM and three or four engineers, so you had these cases where PMs were looking for things to do. At my greatest scale at Uber, before I built the team, I was at about 1 to 25 engineers. And I felt that was actually one of the most productive times for myself and the team, because there was no room for anything but developing product, creating value and moving quickly. No room for processes and all of these different things PMs can get caught up in.
One of the things AI enables is that you can actually go back to that Uber-style ratio and have a PM and three or four engineers, but still create the output of that 25-engineer team. What that means is that PMs will need to shift their mindset to be able to handle that level of scale. That looks like being able to context switch very quickly and being able to manage a number of roles beyond the traditional PM job description. We didn’t have product marketing support. We didn’t have all the design support we needed. So the PM becomes that person who needs to fill in the gaps when you have so many people producing.
And you need to stay ahead of that level of output in a way that still feels cohesive, so it still feels like you’re building something that’s greater than the sum of its parts. That can be particularly challenging when you have 25 engineers, or an AI-enabled team of four or five engineers, because the output is so fast and you’re just trying to hold on for dear life and keep up. That’s one of the biggest challenges facing modern PMs: being able to stay ahead, and to keep the features you’re launching cohesive and additive, and not just, “Hey, we’re shipping everything a customer requests, and we’re doing it as quickly as we can.”
Getting out of the process trap (43:02)
Aakash: Can you break that down in more detail for me? What are the processes PMs used to get caught up in? How do you say no to those processes? A lot of times when I was working at a bigger company, I almost felt trapped by the process. I could recognize it, but I didn’t know how to get out of it.
Greg: I think often it’s meetings. You have to get approvals, you have to jump through a lot of hoops, you have to get a bunch of stakeholders aligned, and you have to check some boxes for planning processes at the corporate level. Obviously, when you have this torrent of productivity coming at you from the engineering standpoint, you have to get creative. You have to be able to push back on the meetings that are just busy work and craft. And for the others, you need to find ways to spend less time on them, and I think AI enables this in a much more scalable way, by automating more of that process of getting ready for and completing those meetings.
What PMs could get caught up in, in the previous world, is that all of that crust, all of those processes, all of those meetings were the actual job, and not actually building product, talking to customers and releasing product. It was, “Hey, I’m going to jump through the corporate hoops. I’m going to check all the boxes on this playbook for how PMs operate.” And I think that’s being forced to shift away in this new era, where people can produce so quickly.
The right PM-to-engineer ratio (44:49)
Aakash: And then you made this provocative claim around ratios. So just let me understand this. What do you think is the right ratio? We can assume the future will adjust toward whatever the right ratio is.
Greg: I think now a team of three or four engineers can produce the output of what a previous team of 20 or 25 engineers could. So a really small pod of, let’s say, three to five engineers at max and a PM can be a good ratio, because they’re able to iterate and produce at a much higher velocity. You gain all of the benefits of being a smaller team and being very closely aligned, without the cost of losing a lot of horsepower like you did in the previous world.
AI for your team vs AI for your customers (45:45)
Aakash: We touched on this a little bit in the podcast, but I’d love to get your take as somebody who’s building an AI company day in, day out. What is the difference between AI that’s ready for an internal audience and AI that’s ready to scalably engage with customers, and how do you build that?
Greg: I think it’s about the technology you build around quality specifically. Whether it’s an internal audience, or even a tool that’s meant to generate outputs you then review, the quality bar needs to be so much lower, because you can feel confident a human is going to look at this before it gets shared in any sort of public way.
But when you strip that away, you need to have the appropriate guardrails in place. You need to be thinking about every update you make and measuring it against both those guardrails and some continued level of human QA. Something as simple as bumping the version of a model: when you’re building something that’s internal, or not directly going to customers, you can just do that and feel pretty confident it’s going to work.
When you’re actively shipping something to customers, you need a much more rigorous process. You need an environment where you have sample data you trust, where you can run these generations on that sample data outside of production and make sure they’re still passing guardrails at a similar rate to before. You do some additional QA to make sure no regression happened that you weren’t expecting. And then, and only then, can you feel confident upgrading that model.
When you’re doing that hundreds of times a week, because you’re updating, improving and optimizing these customer outputs across a complex customer base and trying to optimize for things like conversion, you need to be able to run those sandbox-environment tests at super high scale and with a great degree of confidence. That’s just not necessary when you’re not directly talking with customers.
How testing changes for probabilistic products (48:11)
Aakash: What’s interesting there is that you used fairly traditional PM language around it: guardrails, testing environments, sandbox environments. You didn’t put a new label on the skill, like, “Oh, you need to learn this type of online eval or this type of method gradation. What would you say are the new skills you’ve had to learn, going from the pre-AI building world to now building AI products 24/7?
Greg: I don’t think the fundamentals actually change that much. What obviously changes when you’re dealing with a probabilistic tool is the type of testing you’re doing in those sandbox environments, and the types of guardrails and evals you’re running. Those change pretty drastically. You go from tests your software team writes, even the most advanced ones, that ensure things happen as you’d expect when you interact with the product. Now you’re in a world where you need to be testing the nth possible iteration of how someone might interact with this, and feel confident about what’s going to come out the other end.
So it ends up being a combination of those traditional tests you’d run, deterministic tests that say, “Hey, is this output what I expected?” in a discretely measurable sense, and these more advanced LLM-as-a-judge evals that apply a greater degree of reasoning to a much more complex problem, like groundedness, or ensuring the AI doesn’t do anything it wasn’t explicitly given permission to do.
Those are cases where the AI really wants to please. When customers ask for things… there are stories about this from the early days. I forget if it was Toyota or someone, but someone got the AI to say they could get a free truck on their chat. Those are much more challenging, because the AI and the LLM-as-a-judge both want to please. They want to do what they’re being asked. So you need to think about ways to tie those things back to something much more concrete, saying, “I want you to actually tease apart every specific action the AI has taken in this response, and I want you to link it back to the specific instruction that gave it permission to do that.” It’s not just, “Tell me if the AI has done anything it’s not supposed to.” You have to go a significant layer deeper than that if you want to do this at significant scale.
Aakash: So examining the specific traces where things go wrong is really important.
Greg: Yeah, I think that’s particularly important. And I think examining the specific traces where things go right matters too. You have the opportunity where the AI is being creative and doing things in interesting ways. Take the example I gave earlier. Say there’s a topic where almost no one ever converts after they bring it up, and one time the AI’s response helps them convert. What can you learn from the approach the AI took to get to that point? That’s an equally big opportunity as, “Hey, the AI made a huge mistake here. What can we learn, and how can you tie that back to how it specifically made that mistake?”
How an AI-native startup builds (52:03)
Aakash: In our pre-call, we were talking about how you guys work, and I think this is going to be really illustrative and insightful for others, because you’re in a unique place. You’re not at Uber with 200 other PMs. You’re a small startup. You get to operate and define your processes to the level the current tooling allows. So what are the interesting things we can learn about what an AI-native startup looks like on the inside?
Greg: For sure. A lot of what we’ve discovered is that it’s really easy to build. It’s really easy to build quickly. It’s also pretty easy to build with pretty good quality. Those are things you can get really good at if you have the proper engineering workflows leveraging the latest AI tooling. What’s hard is still building the right thing. So that’s where you spend a lot more of your time, and we’ve found it really valuable to spend a lot of that time in discussion.
There are four of us on the team. Let’s all sit down and talk through this problem space. Let’s have someone bring to the table an artifact that lays out the problem and lays out the opportunity, and then let’s work through it. What are the biggest challenges we need to solve? How are we going to solve them? What’s the experience we want to create? We whiteboard that as well.
Then you end up with an artifact, which is a detailed transcript you’ve recorded from this discussion and a diagram, either on a live whiteboard or a FigJam or something similar. You can feed that into an AI skill that takes it, defines a much more detailed PRD from it, and creates the appropriate Linear tasks to project-manage it all out into the appropriate work streams. It actually has the agents start churning on that where it makes sense, and it follows on with areas where we need to do more definition or have more discussion. So you end up spending a lot less time directly developing, or sitting down at a keyboard and writing out a PRD, and a lot more time actually having those real, substantive debates about how the product should work, how it should evolve and what comes next.
Aakash: So am I hearing this right? Instead of spending your time writing PRDs, or writing front-end PRs as an engineer, engineers and PMs are spending more of their time in meetings sketching out what it should look like, making sure the transcript has that information and they have pictures with that information. Then the AI comes back with, “Okay, here’s a working prototype,” and you improve it and ship it from there.
Greg: That’s right. And that’s why it’s been very important for us, and I think in the future it’s very important, to have everyone at that table be capable, at least to a certain degree, from a product and a business standpoint. We want people who understand what we’re trying to accomplish as a business. We want people who have some level of product intuition. When you have those disparate expertises but at least the same baseline of understanding, that’s where those discussions can be really powerful.
What a big company can borrow (55:40)
Aakash: So if you’re at a larger company like Uber or Bolt and you can’t adopt all of that, what can you adopt? Are there specific tools in how you’re engineering your harness, or specific processes in how you’re working with your product team, that let you operate a little more like this nimble, AI-native new way?
Greg: I think you can adopt a lot of it. Even if the expected output is a standardized PRD that you then need to get approval and review on from different parts of the organization, you can still model how you get to that PRD in this more AI-forward, AI-native way. You can still work with, maybe not your full team if it’s a larger team, but a subset of the people you feel are most important in making those types of definitions and decisions. Have those conversations, and then have the automation and the skills to automatically turn that into the format you need for the corporate processes you have to run through.
So I think a lot of it is very usable. You just need to adapt it to the outputs expected from you as a team, and to the personnel you have around you and what types of people are going to be most effective in those more open-ended, project-oriented conversations.
Where to find Greg (57:12)
Aakash: All right, guys. Greg Greiner is the name. LinkedIn is where you can find him on social media. Woven is his startup, if you want to increase conversion rates or get closer to your customers and not vibe code this whole thing out, but choose a solution that has built it for you. You can check it out. Thanks for being on the podcast, Greg.
Greg: Of course. Thanks for having me.