Check out the conversation on Apple, Spotify and YouTube.
Introduction (0:00)
Aakash: Last year, about 10% of the posted PM jobs were AI PM jobs. This year, it’s double that, nearly 20%. What do you think of this data?
Todd: Well, what we’re talking about is you’re measuring job postings. That’s two things. It’s a company saying what type of person they want slash what type of job listing is going to attract certain types of people. Here’s a word of caution to all of you out there who are hearing this and seeing this. You better damn well be good and know what you’re talking about if you’re gonna call yourself an AI PM.
Aakash: Todd Olson spent 28 years in product management. Now he is leading one of the leading product management tool companies out there, so he has a unique vantage point to help you learn step by step how to become an AI product manager and what it takes to become a CPO or a CEO in AI.
Todd: I think the more firsthand experience you have touching all this technology, the better. This is a real issue that a lot of people don’t think of. It starts fundamentally with solving hard problems.
Aakash: We’re having the CEO of a $2.5 billion company demystify AI PM for you. What’s hype, what’s real?
Really quickly, I think a crazy stat is that more than 50% of you listening are not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify podcasts, my commitment to you is that we’ll continue to make this content better and better. And now on to today’s episode.
What We’re Covering Today (1:29)
Aakash: AI product management is all the hype right now. Everybody’s talking about it. When I go to universities and talk to students, I hear more about AI product management than product management. So I brought in one of the veterans of the product management industry. He has been in product management for over 28 years. He rose to VP of product at a public company, and then he started one of the world’s leading product management platforms, Pendo.io.
Pendo started as an analytics and guides company. Now it is a complete product management platform working with some of the world’s top brands, from American Cancer Society to Zendesk, A to Z, they have them all. And so Todd Olson has one of the most in-depth views into how the world of product management is changing. What actually AI product management means within the context of the larger product management landscape, how to become an AI product manager. I don’t think there’s anyone better in the world who could provide us this vantage point. So Todd, welcome to the podcast.
Todd: It’s great to be here. When you say veteran, it just makes me feel old, but it’s all good.
The AI PM Market Explosion (3:24)
Aakash: So, as I said, the hype is around AI product management. When I talk to anybody about the field who hasn’t been in the field, they don’t actually talk to me about road maps and strategy, the things that I used to hear about, they talk to me about AI, AI, AI. And so I was trying to demystify this myself. And what I did is I went on LinkedIn, and I searched product management jobs last August, and then I searched it this August, and I broke it down into growth product management, core product management, and of course AI product management, and the numbers really astounded me when last year, about 10% of the posted PM jobs were AI PM jobs, this year it’s double that, nearly 20%. What do you think of this data?
Todd: Yeah, well, look, I think it doesn’t surprise me, I guess that’s the first thing I think of. I guess what, you know, what we’re talking about is you’re measuring job postings, so that’s two things. It’s a company saying what type of person they want, slash what type of job listing is going to attract certain types of people.
So really, I mean, if you think about it, this is a marketing game between companies and perspective employees, and both are trying to maximize both sides of this marketing game. So, look, AI is, you could have just taken out PM and just said everything AI and everything you said at the onset of this would be relevant, because the truth is AI’s hot. AI companies are hot, the AI economy is hot, therefore, like the PM flavor of it of course is hot.
Yeah, so, yeah, it doesn’t surprise me. Now the line between AI PM and other PMs, I think is an interesting question. I mean, it implies, of course, there’s some level of like AI fluency, like I understand the tools, I play around with it, I use it regularly, I understand maybe the technology stack, how it works, I’ve used a bit, but let’s be honest. All this stuff is so new, you only have so much experience, like only so much experience, and it’s changing, like literally every week.
And one of the most fascinating things is, and we’ll probably talk about it, but we’ve done two acquisitions in the last roughly 12-ish months, maybe 14 months. Both AI companies, and one of the criteria that we were looking for is we wanted a company post ChatGPT to acquire, not pre. So why? It’s because most companies pre-ChatGPT had to rewrite their entire application. They had to relearn all their own skills. There was like a sense that if somehow you’re just too early, early, mind you, that you’re obsolete, or your skills were less relevant, maybe that’s a better way of thinking.
So sometimes like having more recency of technology is actually better for you in terms of your success rate because you’re learning things for the first time. So I don’t know, like I think it’s an interesting trend. Do I think this is gonna last? I don’t know. But nearly every company I talked to is doing some AI work. And there’s no way that they’re only relying on AI PMs to do it. Obviously normal PMs are also doing AI, so yeah, I mean it’d be interesting to see what long term wins out here.
Why PMs Need to Upskill into AI (6:07)
Aakash: To that point, normal PMs also building AI. It seems more important than ever, if you are a PM to upskill into these AI features, even if there isn’t an AI feature on your roadmap for the next 3 or 6 months.
Todd: Well, yeah, and look, there’s two sides of AI generally speaking for all of us. It’s like one, how we use AI to run, I mean, like, you can see our lives, but like certainly our professional lives, so our work life, and there’s two, is how do we incorporate AI into the products we build to serve our end users, right? Those are two separate thoughts.
Like, yes, if you are not using a solution to do prototyping, like a Lovable or a Bolt or a V0 or a Replit, like you’re gonna be left behind, because other PMs are, right? And it’s really about speed and the ability to just not wait on other individuals to help get to a prototype, help go do discovery, help validate it with end users.
I think if I’m doing market research on a market, if you’re not using like deep research within ChatGPT to just do high level cursory competitive analysis, give me a lay of the landscape, like that is the way you should be working, period. Regardless of what you’re building, regardless of what you call yourself.
And then there’s respect to how do I use AI. It’s all available via APIs now. It’s an API you can call, right? So I don’t care what you’re building, but if you’re dealing with any level of text, you should be using large language models. I don’t know if that makes you an AI PM, but you should be using it. Like you’d be kind of dumb not to. And even things like generating an email, using LLMs gonna be a better email than you try to do it yourself, right?
So I do think that the way we think about it, most of our PMs, regardless of what they’re working on, are using AI in some regards in their customer facing features. And does that mean every single solitary feature has it? No. Like I was just messing around, we’re releasing, or maybe there is a little AI, but we’re just shipping this dark mode for guides. It’s like, you know, we have our in-app messages, you can toggle them, I think AI actually does generate the CSS from the existing CSS, so I do think it’s actually AI generated, but so it’s funny, I was trying to pick something that wasn’t AI related and somehow magically was because nearly everything is, but not every feature is gonna have AI in it.
And to be honest, we’re not marketing it as an AI feature, it’s not AI dark mode, it’s just dark mode. It just happens that under the covers, that’s one of the technologies we use because it’s just a better way to do it.
Why AI PMs Get Paid 30-40% More (9:32)
Aakash: Mind blown, so many insights packed into just a few minutes there. I wanna just re-emphasize a couple points for folks. You may not call yourself an AI PM. We’re gonna talk about that in a second because there might be a reason you want to call yourself an AI PM. And then the second point around, even if you aren’t building an AI feature, you should be thinking about is gonna hit an LLM API gonna enhance the feature that I’m building, even if we don’t market it as such. I think that’s such a phenomenal insight.
So let’s talk about this point you made around calling yourself an AI PM. A lot of people I talked to want to call themselves an AI PM because it seems like AI PMs get paid more. Why is this happening? Is it simply a matter of putting that branding label on yourself? What is going on here? Why is this data showing that AI PMs are getting paid 30 to 40% more than regular PMs?
Todd: Because AI is hot. The market’s hot. Why are AI companies valued greater than other companies? Some of it’s due to growth rate, but yeah, I think that’s why. I think there is also this sort of idea that in an AI world where it’s like everything’s around labor arbitrage, that we’re gonna have fewer people, they’re gonna do more, henceforth they should be paid more as well. Like if I’m doing the work of 1.5 PMs, should be paid 1.5 PM salaries. I don’t know, maybe, maybe I can make that case.
Now my argument would be, all of us should be working that way, so there shouldn’t be some situation where you’re somehow 50% more productive than I am, but I think maybe right now for the time being, there is sort of this disparity between folks using AI technology and the ones who do not, and or understand it more.
But look, I mean, if you were a PM for like a really sophisticated technology in the past, would you be paid more than one that wasn’t? Yeah, probably. I mean it comes down to scarcity of skill set. If I’m looking for someone who I know has used all the various AI tooling versus one that’s never had that experience, and I’m willing to pay a little bit more for someone who has that, like, yeah.
Even us, like we’re an analytics platform. I’ll give you a different example. We have analytics PMs, these are people that know about statistics and are pretty deep technically on data, some of them have advanced degrees in data science. Are they scarcer, more highly skilled, honestly more paid than ones who are just more UI oriented PMs? Yeah, I’d probably say that.
So even technical PMs are just, they cost more because often if you’re a technical PM maybe you have this alternative to be a software engineer, which has its own sort of career pathing with it, so yeah, look, one, it’s hot, so yes, it’s going to be more expensive. Two, there’s scarcity because of the newness of the technology. So yeah, none of that surprised me, and yeah, does that mean you want to call yourself an AI PM? Yeah, it probably does.
Here’s a word of caution to all of you out there who are hearing this and seeing this. You better damn well be good and know what you’re talking about if you’re gonna call yourself an AI PM because I can tell you if I’m gonna be paying any percent more for that skill, we are going to interrogate the hell out of it, right, because there’s so much AI washing in companies now, you know, people calling themselves AI companies when they’re not, or people saying they use AI when they’re not.
I am sure there’s gonna be AI washing on resumes. I’m an AI PM, I’m an AI this, they put some crappy two hour certificate programme that they went to that somehow MIT threw out, and all of a sudden that means they went to MIT, which, by the way, they didn’t, they went to some state school and they’re calling themselves an MIT grad online. So you’re gonna get all that, right? And so don’t be that person. Because all of us can smell through it, we’ve seen this before, you better damn well know what you’re talking about, because if you’re gonna try to catch this wave, so that’s my take at least.
Reforge Sponsor Message (12:50)
Aakash: Yeah, I hope you’re enjoying this master class on AI product management with Todd Olson. Today’s episode is brought to you by Reforge. You know that feeling when you try to prototype something with AI and it spits out something completely generic? The problem is that most AI app builders aren’t built for product teams, for product managers like me and you. They’re built for founders who are starting from scratch, but product teams aren’t building from zero. You have an existing product, real customers, design guidelines.
Reforge’s new product, Reforge Build, generates prototypes that reflect your real pricing tiers, real features, real customer language, not generic placeholder. So stop fighting tools built for founders. Start using a tool purpose-built for product managers to AI prototype well in the way that we’re teaching this episode with Todd. Try it for free at reforge.com/Akash. That’s reforge.com/AAKASH and use the code BUILD for one free month of premium.
The Real Requirement: Production at Scale (13:54)
Aakash: The people who are fibbing on their resume, they aren’t landing these jobs. All these jobs are pretty rigorous processes, and in general what I found is that if you’re gonna want to land one of these AI PM jobs, you would have had production at scale successful AI feature experience and all three elements of those are key. It’s not production successful at a one person startup that you created last month, but production at scale with 20,000 paying B2B customers. Now that’s what they’re looking for, right? And so for a lot of people, it’s going to be, how do I develop the experience internally at my own company and my current PM role and doing that via upskilling in order to land one of these roles.
Todd: 100% makes sense.
The 5-Layer Technical Pyramid (16:30)
Aakash: So, that’s what we wanna help you do for the rest of this video, guys. This is a high level view of how to think about if I want to upscale into AI features, maybe, you know, my roadmap right now. I can’t think about the example Todd just gave of tapping into an LLM to check out the CSS. I’m not at the intuitive level of building AI features. You need to work your way up this pyramid.
So you need to start with foundational skills, which we’re gonna dive into for you guys. AI ML fundamentals, data pipeline understanding, and prompt engineering. These are the basics. Then you need to move up into observability and monitoring. How do you do trace analysis? Do you know what trace analysis is, when I just said that, or is only the word debugging ringing a bell? Well then you need to understand debugging for AI. Production monitoring, what that looks like and cost optimization. Pendo has some really cool features we’re even gonna show you to do that.
Then you need to move into evaluation and QA. So it’s funny, like as I got more senior as a PM, I started doing less and less QA, especially once I hit the group product manager, director, product manager. Now I talk to them and they’re back into it, but what are they doing? They’re doing evals. And so what are evals? How you AB test different prompt engineered versions of a fine tuned LLM, how you look at the quality metrics and KPIs, that’s the next level.
And then we get into the level of Todd’s probably operating at product strategy, what is your AI product roadmap? Well, how are you managing your stakeholders, and then your leadership skills, AI ethics and team building and culture. So this is the roadmap, everybody. Let’s start with AI ML fundamentals. Todd, what should people be focusing on when it comes to the fundamentals?
AI/ML Fundamentals (17:26)
Todd: Yeah, I mean, look, you need to know how it works. You know the core and what it does, what it’s capable of. Honestly, you seem to play around with it a lot. So I think the more firsthand experience you have touching all this technology, the better. I mean, you have this concept of token economics, yeah, we’ll get to that more around costing and things like that, but you need to start understanding the trade-off decisions between using different models, different levels of different models, like when are you GPT-5, you’re going back to 4, right.
What’s the difference between Anthropic and OpenAI in different use cases or even like Gemini and other ones, I mean. Understanding some of the open source options that are out there and self-hosted options that are out there, like privacy and security is actually a real issue when I talk to customers with respect to AI and maybe that’s another one of these things, but understanding like what’s being trained, where your data is going, all those sorts of things, I think these are all valuable skills, so.
Aakash: I think you brought up a really interesting point around Gemini isn’t even included here, but Gemini is a really interesting model. You should know what is Gemini useful for? In my opinion, you know, one of the things that’s really useful for is it actually understands video. And so anytime I’m building a product that it needs to be truly multi-modal, like I want to input a YouTube video, so I just recently built like a script analyzer of my competitor podcasts. So I built that on Gemini, and so you need to understand what is Gemini useful for. I think another thing you mentioned there is open source, I think it was Brian Chesky of Airbnb just made waves, where he was talking about how they don’t use OpenAI models in production. They’re using a Chinese open source model, I believe, Alibaba’s Qwen. How do people think about that? What is your take on what Brian Chesky said?
Todd: Look, I think you gotta test different models for different applications. There’s performance characteristics, there’s quality characteristics. Maybe you don’t need the quality of GPT-5, but performance speed and cost becomes a much bigger issue. I think with respect to data residency and privacy and what you’re trying to do with your customer base, I think that is a very relevant decision as part of this.
Like we’ve had issues with data residency where certain models aren’t available in certain countries, and we have various data centres across the world that we have to be cognizant of. So, I think these are all things that, if I can say it’s a Pendo hosted open source model, it’s all within our infrastructure, already within our DPA that you’ve already signed and all that sort of stuff, that may be easier for me than convincing them to add OpenAI to my DPA.
Because OpenAI for a while wasn’t even supporting certain countries, the Microsoft version of it, which by the way, isn’t the exact same version that you’re gonna get because they weren’t literally at parity with respect to the, so these are all little details that you may come up with some cool idea, you may go to build and ship it, it may not actually apply to your customer base at the level you expected, and yeah, we’re a huge Google shop, that’s why we know Gemini and Gemini is easier for us to include because Google’s already a provider for us versus adding a new provider.
And if the quality is good in certain use cases, which by the way it wasn’t in some, it wasn’t in others as you were just referring to, like we actually have both, and then we use Claude for a lot of code because it seems to be the best at code generation right now, so we, and they’re constantly changing, so we test against new models all the time, so.
Aakash: So if you’re watching this episode a year from now, all that’s already out of date, and that’s what we mean in this section is stay up to date is part of AI fundamentals. It’s a new skill set for you. What are the newsletters? What are the news sources you’re subscribed to?
Data Pipelines and RAG (23:00)
Aakash: The second area is data pipeline, and I know that when I first published this and sent this to my editor, they were like, this is the first area. They were like, what? Why is this in the foundation? Why do PMs need to know this? I would argue it’s very important. I think that RAG is basically the most important thing that people are building in their product features, which is retrieval augmented generation, I believe, I think I butchered that, but it’s something like that where you hook it up to a vector database in order to get real-time information.
Let’s say Pendo’s building a customer support agent, their support thing on Zendesk or wherever it is, is really good and up to date, so they need to hook that up to there, so that when the customer support agent updates an article, you know, that can be updated in real time, and so that’s why this foundational understanding of data pipelines is so important. Do you agree? Should PMs really be focused on this and what do they need to know.
Todd: Yeah, this is. Yeah, I mean, I think RAG is kind of a de facto way to build it, and it comes back on the previous slide you had context window as one of the terms. So it comes down to how do we supply the context with the right level of context for the LLMs to do their jobs well. So a lot of times what these solutions are doing is you’re passing everything you need in up front, and it’s kind of munching it right there on the fly, but getting the right context to it.
If you put too much in the context window, just like a human, if you give them too much context, you get really confused, so you wanna give the right context. So the way a lot of this works is we’re ingesting data, we’re creating embeddings based on the content we’re ingesting that we’re feeding into a vector database. So when someone asks us a question, we’re then looking up what context is relevant to that question they get passed to the LLM to answer it.
So it’s actually a core way to build, and seeing how it works, what’s the performance considerations, when to do it, I think one of the things that, you know, it’s kind of implied in all this, yeah, you have governance, but also just scale. How do we get these systems fast? I think we, this is the thing where we’re doing acquisitions of AI companies or we’re thinking about AI features. When we launch it up, we support some of the largest web apps in the world use us, it’s just got to work at scale, and if it doesn’t, that’s a problem, if it falls over because it can, everything’s too big, then that’s another thing, I don’t think we have scale in there, but that, but yeah, I think this is huge. I think this is a must-have, I totally agree with you, so.
Aakash: And so this is where Todd was mentioning those technical PMs getting paid more. Part of being an AI PM is being a little bit technical.
Prompt Engineering (26:56)
Aakash: And the final component of the bottom layer is prompt engineering. I think a lot of people are going to roll our eyes at this, Todd, because they’ve seen so many posts on social media. Check out this killer prompt. Check out my prompt framework. Tell me why prompt engineering actually matters.
Todd: Because I, you know, again, it also comes down to a little bit of context and instruction, the better context and instruction we have that we’re setting the LLM, the better response we’re going to get. So I think there is a bit of a skill set to it, but people probably would have laughed at me if I said 10 years ago there’s a skill set in how I use Google Search and I’d go look for things, but probably was, some people are probably more effective than others, and I think this is a similar thing.
So understanding what a good prompt looks like, what level of detail, yeah, I mean, I think this is sort of must have as well for it, and this is actually, it’s interesting you said something earlier. In your first graph, you have this concept of platform PM. So we have platform PMs, I wonder if we’re going to see a world where there’s going to be an AI platform PM which understands the embeddings piece, RAG, maybe some of the core fundamentals and expose services to other PMs which are more domain experts, subject matter experts on the particular vertical maybe or subject matter experts in certain areas, and but those folks are still gonna, everyone’s gonna have to know how to understand prompt engineering.
That’s where I think maybe it’s like, we’re all gonna have to because you’re gonna want a subject matter expert or domain expert in a certain area to use these prompts to teach the LLM how to think about the problem it’s solving, how to use the data you’re passing and etc. etc. so.
Aakash: You guys heard it here. Everybody needs to learn prompt engineering, and I love this point you made about different types of AI PMs. We didn’t talk too much about that, but the role is already specializing, and I think you just forecasted the future where we’re gonna have AI platform PMs, we’re gonna have more end user PMs at the AI companies already, we’re seeing that there’s research PMs and product PMs, so that’s already a very clear separation. We’re gonna start to see more and more bifurcation, I think, as the role grows.
Todd: Yeah, yeah, I think so too.
Trace Analysis and PM-Engineering Tension (33:02)
Aakash: So we’re moving up one layer. As we said, most people, they understand debugging, but they don’t understand trace analysis. What do people need to know about this and why is it important to building AI products?
Todd: Yeah, look, I think if you have more orchestration, you have agents calling other agents, calling tools, really understanding what’s happening behind the scenes because what’s getting sort of passed from agent to agent, I think is useful. Yeah, I mean, performance, it can often get back, you know, high levels of performance, understanding where things break down. Sometimes it’s just errors. So within a large chain, so sort of understanding where was the error and how does it sort of recover because it tries to redo itself and fix itself, there’s a lot of optimization within there.
I think this is interesting. I mean, I think this is one where I do see a sort of a partnership between PM and Dev. This is an area where I’m actually seeing tension amongst teams. I’m having engineers, some engineering managers like no, this is my world. I don’t want some PM sitting next to me shadowing it. So, I want you to tell me what to build and why to build it, not how to build it. I’m gonna make sure it’s performing, I’m gonna make sure it works well, I’m gonna make sure you hit your requirements, which then the evals gonna cover that piece.
So I think this one’s a question mark for me in the sense that, I think at a startup, yes, you’ll be doing this. I think in a large company, there may be a little bit of standoffishness from engineering or some division of labor that may try to exist, maybe not, if you’re best buddies with your engineering leader, you’re going to sit next to him, but you just got to think about a team of 4 to 6 engineers and 1 PM like how does that work with these systems, but they are, I mean it’s, look, first of all, more knowledge is power, so I would definitely know how it works. If I’m talking to an engineer, I want to say, hey, what’s the trace say? Because I can ask the question and get the results back, but maybe you’re not the one actually doing it, so.
Aakash: This is where your vantage point is giving us so much insight. As a PM you need to tread very lightly. You need to learn this skill on your own agents that you’re making for your own personal productivity, but you’re not expecting, I’m gonna be the one deciding what observability platform, whether we buy Arize or BrainTrust. You’re not the person who’s actually gonna be implementing Arize, and then doing the majority of the trace analysis. But you need to be fluent in it, right? And sometimes maybe the feature breaks in production, everybody’s off on a P0 bug. You don’t want to be the PM who’s just there helping coordinate. You want to actually jump in and help, and that might be a point where you go in and you say, oh yep, I checked the traces in Arize. These are some of the errors I’m seeing. Are you guys also seeing this? So it’s almost like you only pitch in a case, you’re not actually doing this work.
Todd: Yeah, 100%. Yeah, I mean, it’s like just like Data Dog. Most PMs don’t make the Data Dog decision. Some don’t even probably have a login to Data Dog. I know no PM wants to be on the pager duty, you know, and where they’re getting called in the middle of the night. So, so. Yeah, it’s an interesting one. I agree though. If you can help out, if you can lend a hand, if you always do so, it always makes you more valuable, the PM, but yeah, this one, yeah, it’s an interesting one.
Aakash: What are the other areas of PM engineering tension that people should be aware of?
Todd: Yeah, I think who owns what is always a level of PM engineering tension. Yeah, PMs or engineers want some autonomy and they also they want to be valued and skilled as well, and they want to be able to play around and experiment with things. I think they should have the right to do so, so there’s a craft to being an engineer as well, right, and we want to sort of respect that craft, so I do think just some level of separation of concerns is valuable and look with AI I think we would all say everything’s sort of smashing together, but a PM’s job is by necessity, a role of influence.
Where a lot of people are influenced by your decision making, but yet do not report to you directly. I mean, that engineer has a boss, and probably a boss’s boss, and that person probably made a decision on what telemetry tool you’re using. It’s not even an engineer, it’s probably multiple levels up, right? And so be respectful that that person has a whole chain, because the engineering organization is so much bigger and there’s a lot more levels there, and it’s a lot more complexity there.
So yeah, I mean, it’s interesting you’re talking about if there’s a P0 related to something, it’s the engineers that are getting pulled in and fixing it, not you. And I think it’s useful for you to be able to see some of these things because bugs do affect your capacity as a PM indirectly, but if a bunch of bugs are coming in, it’s ultimately going to affect your innovation roadmap, so it’s kind of good to understand this balance in your head, because if you take a less quality oriented approach, you’re gonna be the one to get affected by rework and things like that, so, so yeah.
Production Monitoring (37:55)
Aakash: So wise, so production monitoring. This is another one. I’m not sure, where’s the line of the PM here? How deep should they be going into these tools?
Todd: Yeah, I’d say not as much, unless you’re a platform PM and we like to have PMs on platform teams, but how many of our engineers don’t even do this, we have a whole team of ops people, that is their job, and ops, SREs, and they’re really good at their jobs, and there’s a lot more considerations for how the infrastructure runs globally and, but they’re the ones that are detecting this and monitoring it and managing it.
Yeah, I mean, traffic patterns, yeah, if there’s a denial of service attack, it’s our ops team doing, we’re not bothering PMs with it, the head of ops is calling me on a weekend, not our PM teams. There’s not much the PMs can even do, but, well, here’s the other consideration I failed to mention before, but it’s certainly very relevant here. By contract with our customers, only a very few number of people can touch customer related systems and customer related data, so they have to all be background checked, and we don’t wanna make every single person at Pendo get a background check because that feels onerous, right? But the people that do this work all are.
That’s why there is a separation, and so, yeah, if you’re a tiny company, you’re probably able to do this. If you’re at a larger company, heck, yeah, you won’t be, you won’t be near it.
Aakash: So Yeah, there’s this huge difference of what type of PM you are, what size company you’re at, how sophisticated they are. You might even find some of the smaller companies where the SREs, the DevOps teams, they’re handling this. So, be very sensitive to what product management is at your company. We’re talking about product management abstractly, but in the 5 product management jobs I had in my career, it was completely different at each company, and that’s probably the case for every PM out there.
Todd: 100%.
Cost and Performance Optimization (40:44)
Aakash: Cost and performance optimization. I think you guys are on the bleeding edge of thinking about this. What do people need to focus on here?
Todd: Yeah, I mean, this is a real issue that a lot of people don’t think of. We think of it because we deal in the data world, and so there’s just a lot of data and complexity. The truth is how you build and design systems affects your cost of goods sold your COGS which ultimately affect your gross margin, which ultimately affects your success of the business.
Now it’s an interesting time and place because I joke about gross margin, but some of the fastest growing AI companies in the world have very unattractive gross margins right now. So one could argue that it’s trading tokens. So you’re paying tokens to them and they’re paying potentially probably to Anthropic or someone else, which is like this fascinating circle that we have. We have these AI apps, turn tokens, go to the model providers. Those model providers just go to GPUs, or GPUs, just vicious circle, but we, as we know, so why should this matter, because over time, you will have to get to a rational gross margin and we’re going to see radical innovation in this area.
At some point, we’re gonna get diminishing marginal returns on quality. There’s only so good it’s going to have to be, where everything’s going to shift to size, performance, if you listen to Ali Ghodsi, he’ll talk about tiny models as maybe the future. Future instead of large models. So, a lot of people think that the future will be more in just smaller, more tuned models because the only way you’re going to get the gross margin characteristics of a successful, viable long term entity, some of these companies are operating in a sub 15% gross margin, it’s not a business. I mean, no offense, it just is.
So, a typical software company like a Pendo-like company is high 70s, a great company can be in the 80s, so I think that’s why it all matters. And you’re gonna have to whittle it down and, look, I mean, here’s the good news is we’ve seen past this and when Pendo started, its gross margin was much worse than it is now. Why? Because we optimized for speed of innovation than we did for cost, so that means we overspent on a bunch of infrastructure in the early days of the company and over time it became important for us to find efficiencies and now we actually have teams of people to do it.
So this is maybe a warning for all you AI companies out there, they’re just focusing only on speed and growth, which is great, do that now. Eventually you’re gonna have to worry about oh, we got to rearchitect this entire system, it’s gonna take us a year because we got to use different infrastructure, because the infrastructure we picked, well, easy to use and really fast to market, it is just too costly. And that’s kind of what we’ve thankfully gone through a lot of that work today, but yeah, I think everyone’s gonna have to be thinking about that.
And oh, yeah, I love your example because this is often the case. Very often the path to better performance leads to cost savings, almost always. Almost always when something’s faster, it’s cheaper. That’s just a good rule because essentially buying compute, if you think about it, anything at some level goes down to compute. So that’s so it’s always like a twofer. One, none of us like investing in this stuff because we all like building new features for customers. Let’s be honest, most PMs are growth-minded humans. I’m a growth-minded human. But it is cool when you save money and make things faster. Very, very cool.
Aakash: And I’ve noticed this typical pattern when building AI features. Certainly this was the case at Apollo when we were building the email AI writer, for instance. You can hill climb up your evals. And then once you reach a good level, like, oh, in the email writer example for Apollo, humans are accepting the draft that we’re creating as 70% without any edits. That’s amazing level. We were at 20% to start. So we’ve gone from 20% to 70%. Then you can go in and you can start to cost optimize.
And so what our teams, some of our AI research specifically teams did. They tuned the prompt so well, they found a cheaper model that they fine tuned so well, that they were able to hit that same eval result of 70% with much cheaper cost. So sometimes you can also go up the hill and then kind of climb down on cost.
Todd: Yeah, I mean, you mentioned caching strategies too, like what do you cache and what you don’t cache? Do we really need to recompute some cluster algorithm every single time someone comes to a page, no, wildly inefficient and expensive, but a lot of people do it because it’s the fastest route to market, and oh by the way folks, for all of you generating really cool using code generators and oh I use Replit generated my app, they don’t usually take into account the cost savings piece in these generated apps, it just sort of works. So usually all these things need to be rearchitected to think differently, have caching, things like that, so.
Aakash: Yes, especially with anything vibe coded.
Evals: The PM’s Domain (48:56)
Aakash: So we’ve been mentioning evals, we’ve finally got into the topic. I think this is another one where I’d be really interested to understand how deep should the PMs be going? How should they be inputting into the eval process?
Todd: More so than some of the other pieces is the short answer. I mean, I think this is a real area that people should be focused on. Yeah, a lot of it I’m seeing is custom frameworks, but. Unlike automated test suites, which if you go back to automated test suites and engineers would be focusing on unit tests, just fundamentally do these methods work, to integration tests, these sort of systems work together to maybe some level of UI automation tests, and you have testing teams that work alongside your PM teams, but this is a little different, I mean, this is AI grading AI and the quality of the evals, I think matters a lot and the PM is probably the best suited human being to author and manage these sets.
So this is an area where I think it will be a competency, it’ll be a must-have for companies for all PMs whether an AI PM or not, because all of us are going, I do believe, be using LLMs in the future in regards to what we’re building, and I think this is an area where we’re going to have a lot more innovation in this area to make it easier to do this, but yeah, I think this is going to be a hot area, so the PM is the best positioned human.
Aakash: I love that insight here, you are the expert in the user, you’re the expert in what the business needs. When we’re talking about things like trace analysis or some of the other stuff, the engineers are clearly the expert, and so it’s more about how you work with them. This is an area where you need to be the expert.
Todd: Yes, I agree. I agree. Now the engineers are probably got to supply the frameworks, there’s probably going to have to be some harness situations set up, there’s going to be some tooling there, but yes, you are the person that understands how to do this better than the engineer. So I would definitely invest here, prioritize this.
AB Testing and Experimentation (52:35)
Aakash: AB testing and experimentation. It’s a classic topic. You guys helped develop this field for our product managers to begin with over the last decade plus. What do they need to know about it when it comes to AI?
Todd: I don’t think it’s a significant difference than what it normally would be. You know, you, yeah, I don’t know what statistical significance is. There’s a variety of different tests. There’s split tests. There’s classic AB, ABC tests. There’s different ways to test depending on your user base. This will also depend on whether you’re a B2C product or a B2B product because that’ll determine the end, the number of test subjects essentially you’re looking at, but basic statistics.
Yeah, and you’re gonna test a variety of things, just like you test a UI or a button, you’re gonna test a prompt, you’re gonna test an LLM provider, you’re gonna test different things. Yeah, I think this is basic stuff. So I don’t think this is particularly complicated, but I think in the world of AI it’s really useful. What makes it exciting, what makes this more relevant is that, let’s say you weren’t doing AB testing experimentation before. In a world of AI, the cost to generate a variant is much, much lower.
So, so. You wouldn’t have tested as much in the past, and I think that’s the interesting mindset shift. We’ll probably talk about discovery a little bit later, I still think discovery is critically important, but maybe in the olden days you would spend a little more time in discovery and a little less time experimenting because it’s just so damn costly to generate more variants, whereas now, hell with it, get something out there, run a bunch of experiments, and iterate your way to success. I mean for some products it’s gonna be an OK strategy, just depends on your user base, but I think this is pretty interesting, so.
Aakash: There were so many core product teams I’ve been on or managed where we had some experiments on the roadmap, but as the realities of building things came, we ended up maybe launching a couple experiments a quarter instead of the 12 to 15 we promised at the beginning of the quarter. I feel like with AI reducing how much time it takes to create, especially these front end only changes, there’s really no excuse for PM teams to go entire quarters and months without experimenting.
Todd: Yeah, I think, the really hot take on this slide is that there’s something new with respect to AI and experimentation. I think what’s new is, if you were not doing it before, you probably have to do it now, and that’s the new thing, a lot of people could avoid this in their day to day jobs. They didn’t need to know how to do AB testing or experimentation, because they never had the capacity to do so. I think all of us are gonna have, that’ll be a tool in all of our tool bags that we’re just gonna have to exercise more and more than we ever have before.
Aakash: And the good news is, I made my entire career on these small little growth experiments, so they will work for you guys. Still probably the biggest dollar impact change I ever made in my career was changing the search ghost text at Threadup.com, like these little things can really give you a huge impact. So I think you guys will all benefit a lot from putting these little things into your roadmap with AI.
Quality Metrics and KPIs (56:03)
Aakash: Let’s move into quality metrics and KPIs. How does the metrics understanding that a PM needs to have change for these AI features?
Todd: Well, we talked a lot about the costing implications and things like that, I think, look, I think the leading indicators may be a little different. Like for example, if you’re doing an autonomous agent to do a bunch of work for folks. Do you care if you have daily active users? Like, no, maybe not, right?
So I think what’s going to shift is, we’re going to care a lot more about outcomes. And by the way, we always should have been caring about outcomes, but I think the daily, weekly active use is gonna for some products be sort of less important. Now for some products may be more important, if you want the end user to be interacting with your agent a ton, then that obviously is a really important metric for you, but if you don’t care, you just want to get something done for the user, which is a lot of B2B products, I think it’s going to be a lot more shifting towards that.
And I mean one of the examples I love from the AI world and the B2B world is, you know, Finn, which is this customer experience agent, has this kind of 99 cents per support ticket closed, doesn’t matter whether the human touched it or not, that’s the kind of models that are super interesting to me. Now, look, not every product has something as clean as support ticket closed, that’s very clean, unambiguous outcomes, but I think, we all have some outcomes we’re trying to drive, so I think there’s a, that’s what we all should be thinking about. What’s the goal of your product? What’s success look like and how do we make sure we’re measuring that?
Aakash: This seems like one of the ones where that’s why I didn’t put any time to proficiency here. It’s less about I need to go study something for 3 months just to upskill. It’s actually more applying your common sense, your logic, but to these new sets of features, and the way you improve this is either by shipping those features at your company. But if you don’t have access to that, think about how would I have measured the change that somebody shipped recently. Notion launched AI agents. It seemed to get a lot of hype. Maybe you tried it out. How would you measure that? Because as you walk through that process, you realize, oh wow, measuring this agent isn’t the same as the other features. And so I think that’s the main thing here.
Todd: Yeah, and we’ll talk more about this and hopefully I’ll show a little bit of it, but we’re looking at what are they using the agent for? How good the retention rate is, we’re looking at things like frustration signals right now for agents, are people pissed off? How often does someone have a short pithy one line response, just do this for me, because that’s if I’m irritated when using something, then that’s a negative sentiment.
So I think, I actually I think, in this AI world. I don’t know if we’re gonna talk about it on one of the little blocks here, but I think being close to the customer is even more important. And that’s why you’re hearing about this forward deployed engineer, or we’re talking here at Pendo forward deployed PMs where we go on site, sit alongside our customers, and we tweak the context window, we tweak what sort of embeddings we can pull back, so go back to your core PM skills, of course you have to know those in order to sit on site with a customer and tweak, but we wanna tweak, we wanna obsess over some workflow and say we’re gonna automate this whole thing, so it is flawless for you.
But we’re gonna send you on site to go do it, and their successes like successful workflow completion, that would have taken this user hours down to seconds or minutes, that is a success measure, and then, but yeah, it’s and how satisfied is the user? Does it actually replace you doing it yourself? I think that’s the kind, that’s where we’re going to as an industry, it’s kind of exciting, to me it feels like far more transformative than I’m gonna create this page with a bunch of table and data cells and some forms and stuff, right? It’s how do we make sure we obsess over this workflow and fully automate it. That’s huge.
Aakash: I love this idea of a forward deployed product manager. I want to do a quick plug for one of yours, I believe, Dave Colleen, field CPO. He has this amazing podcast. I think that’s the type of thing that is what’s gonna happen.
AI Product Roadmap: Solve Hard Problems (1:01:12)
Aakash: So. Now we’re moving up. I think part of road mapping is discovery. This is, I think, probably the closest bucket I put for feature prioritization. So how do people think about this? A lot of teams, they want to just build the shiniest object, right? They have the shiny object syndrome around AI that, OK, well, now that Claude 3.5 is the new thing, now that Claude MCP is the new thing, now that AI agents are the new thing, and they’re just running after each shiny object. How do you actually build a good AI product roadmap?
Todd: Yeah, look, I think it starts fundamentally with solving hard problems. This is where this whole AI washing can come into effect. I mean, there’s a lot of times where I think about building something and I’m like, are we gonna do a much better job than ChatGPT out of the box? Why would we just wrap that, put slap a Pendo logo on it and ship it to a customer? No.
And so then it becomes, OK, well, what unique assets, data, context do we have that we could provide that would add something like a step level above what we’re doing. And go back to what problem are you actually solving for end users? So, some of our early bots, I’ll show some of them, but we’re talking about discovery. What’s the part of discovery that’s hard? Honestly, just finding which damn customers to go interview and prioritizing and setting up all the interviews, that’s the painful part.
You think PM wants to be a scheduler too? No. One, you gotta figure out who are good people to talk to, for what reasons, you need a thesis around it, you need to go get on people’s schedules, so one of our first takes is automating that entire workflow, so it just gets done for you. That to me sounds valuable. It’s a hard problem, and it would take literally weeks of work to do, so that’s the types of things that I focus on is let’s look at something that’s a hard problem, it’s tedious, no one likes doing it, and let’s really obsess there. So yeah, that’s kind of how we think of all these AI features and bets is it just it fundamentally has to go back to that, and it can’t just be AI for AI’s sake.
Oh, we can redo this text. I mean, I always tell teams if I’m just cutting and pasting something and throwing in ChatGPT and throwing it back on our product, just let them do that versus that’s not a game changing feature. So, so I think a lot about that, and then, you know, I like what you have around tech debt here, it’s be willing to throw things away that aren’t working or that are already obsolete. This technology is changing so much that you may try something and you may realize it doesn’t work and you’re gonna throw it away. Do not hold on to it, throw it away.
Too often we hold on to something, we shipped a few features couple like a year and a half ago that weren’t great and we just turned them off. Turn them off, they’re not great, they don’t solve a hard problem, people really aren’t using them, turn them off, be unafraid. Because the more stuff you have in your product, the worse the experience is, just by default, right? So I think we have to be really vigilant here, how good is this thing? What’s the quality of the problem it’s solving? What’s the retention rate for it and go from there.
And then, yeah, I think having a point of view is really valuable. You have this competitive positioning, strategic technology events. What’s Pendo’s AI story versus other companies? Are we just following everyone else? Everyone has an agent, we have an agent, I don’t want to follow people. I think you’ll see that we’ve taken a different approach in shipping agents per se that we have this agent mode. We see it more as a modality, we’re using our products differently versus individual agents.
My vision was, some companies talking about digital workers, the AI PM. I fundamentally think org charts are going to be less important in the future. That’s my vision. I think what matters in business is series of workflows. How do we get something from idea to customer, that’s a workflow, right? And how we operate across that workflow, it’s gonna be a set of humans and agents working together, coding agents, discovery agents, maybe PRD writing agents, but it’s a workflow that really matters.
You think about your recruiting in your business, those are key workflows, and some of these things, you need humans involved. Would you take a job, Aakash, if you literally never met a single person face to face? Of course not. No, and we hear these horror stories of people hiring people without meeting them face to face, and it turns out that they’re working 6 jobs and whatever, right? So it doesn’t pan out in that direction either, right? But would I hire an agent to sift through 1000 resumes that I got through my inbound portal? Yeah, I don’t like doing that.
So again, these are all workflows that those are really what matter and roles are going to blend. But that’s our positioning in the market against other folks. Some of our competitors are gonna create agents and they are gonna put titles on them, I know they are, and that’s fine, that’s their vision for their future. Our vision’s gonna be different, we’re just taking a different tact, and but it’s all based on, and I think customers then self select the companies they wanna work with, based on the alignment of vision.
And that to me, all great companies have a strong point of view, and ours has always been, we’re a platform first, and we wanted, and we went very wide as a company very early while everyone else went deep. But now we’re probably a combination of deep and wide across different areas, but yeah, our AI strategy is very wide. These use cases, workflows crosscut our solutions, what’s fun is, what was defined as a product, we kind of eliminate the lines, which by the way, is a really hard problem.
When you had a product manager owned a product, and now you’re telling them to own a workflow that cross-cuts 4 products, that’s a different skill. Different technology, different stacks you gotta learn, so but that’s to me if you solve that and nail that well, you’re gonna have an amazing experience, so.
Aakash: Yeah, this is super messy to actually execute on the ground. There were so many insights in there, but one I would re-emphasize for you guys is killing those bad features. Nothing is gonna kill your AI adoption of your new AI feature than a really bad AI feature that’s existing in their face. So you need to kill those features, but that might mean, if you’re the PM on that feature, you need to go up to your product leader and say, hey, the retention of this feature is bad enough, we’re at 10%, 4 month retention, it’s time to kill this feature. You’re gonna be the one closest to that data, so you’re gonna have to proactively work around it.
And sometimes when you’re solving these workflows, you might not have a core surface area that you’re owning in the product that you’re so comfortable and used to. So, with AI you need to be really thinking next level about what are these problems, what are these workflows? What does this mean for what I should focus on? Should I really own the surface area or not? Some PMs ask that question, most, they struggle with it.
Live Demo: Pendo’s AI Features (1:04:16)
Aakash: So that concludes our masterclass in AI product leadership. Todd, can you now show us Pendo’s AI features live so we can see how you guys are shipping AI at scale?
Todd: Yeah, so this is Pendo, I’m basic homepage logging in. The first thing I want to talk about is, we’re talking about how to measure success of AI agents, we talked a little bit about sort of, we talked about evals, we talked about observability and traces, and we talked about things, but what we haven’t talked about is just what’s the user experiencing and how are people using it.
That’s an area that we’ve been investing in. So, we have a new type of analytics that we’re calling agent analytics, and if you kind of come in here, you can see, let me just work into an agent. So imagine my company, I have a variety of agents, some of which are customer facing, some of which are employee facing, and I just want to get data on how people are using it.
I can kind of log in, I can see number of conversations, number of prompts, unique visitors, growth, overall retention, I can see use cases, so it’ll actually use topics and themes and start grouping and organizing conversations. You can see if I’ve, if it’s a B2B product, I have customers, I have users, I have retention rates, retention rates are, hey, do people who ask this type of question and come back and ask it again, which is a pretty good indication of quality.
I have some here that was 0%, success rates, so you can see here the way it works. I also have this concept of rage prompts, which just like we had rage clicks in the replay days, we have the same concept here, and if I kind of look at some of these rage prompts, why can’t you find my confirmation number, you can kind of even click right into the conversation, you can kind of see the information of what’s happening, and then you can actually go and watch a replay as well, so you can kind of see what the user’s doing while they’re doing it.
So, this also sort of highlights sort of the connectedness of our platform, I told you earlier that part of our view on building any features is taking a platform first approach, but here’s something where we have integration with AI agents, it leverages our existing install, it’s capturing full conversations, and it’s giving some level of success metrics down to the conversation level of what’s going on and of course, this you can slice and dice and group by segments, group by accounts, etc.
And then the other thing we have is we have the concept of analyzing paths. This is really powerful in that you can actually show what activity was happening prior to getting to the AI agent, so this is, I’m just gonna quickly run this, but all these AI agents as people are rolling them out in their software, it’s in the context of existing software. So, a lot of this is what we’re calling hybrid applications. People are gonna have situations and I think the key will be how do I optimize the experience across these various paths.
Yeah, so you can see here it’s running and I’ll let it run. But the general view here is, this will give you context between sort of traditional UIs and non-traditional, and you can see here just a very quick path, you can see how people got to it, you can watch replays along here to watch how people traverse the application. So, in a world where you’re introducing a new concept, a new UI element, you want to understand how people getting to it, where are they going, this is a really good way to visualize it.
So that’s kind of one piece I want to focus on, I think the other piece, I guess since we’re talking about hybrid, let’s jump into our dashboards because this is also an area where I think it’s useful to see how AI can be used in the context of broader applications, so I showed sort of a path. These are funnels, and part of this idea is OK, I have a thesis as a product manager that we introduce an AI assistant that is going to improve conversion of certain outcomes. We talked about outcomes earlier, and I can see complete booking before AI assistant and see it’s 9%. Well now I have an AI assistant confirmed book, it’s 46%.
So this is a good way, we’re also talking about how do we talk, tell a story to a higher executive audience. What’s the ROI of investing in AI? Well, it’s improvement of people actually confirming a booking. That’s the improvement, that’s the ROI. And you can see here, from there, this dashboard also just talks about generally speaking, success for our product.
So in the way we think of dashboards is, I want systems that are self-service for my team, so if I have a question on this AI assistant, I’m not gonna slack someone, I’m not gonna call a meeting for someone, that’s a waste of my time and their time. I’m gonna come here and I can see high level metrics for this, and we saw some of this earlier, prompts, conversations, retention. I have a goal set up for the assistant, so I can see how we’re tracking against the goal, I have use cases, that we also talked about, these are sort of the emergent use cases.
I have information if I want about the team, the designer, the product manager, the program manager, so this is like one stop shopping, so if I have a question around this, I know who to go talk with, I have a high level platform approach, I think you’re talking about, you’re bringing in so many different elements here.
Aakash: Exactly, it’s qualitative, it’s quantitative, I can see weekly active visitors growing and it actually talks a little bit about what it is and why it matters. User adoption of the overall assistant, this is just a good solid PM dashboard. That kind of looks at all aspects of what’s going on, so. If I see something I like, let me go back to this chart, I can actually ask a question right in line, why did this dip, and it’ll go off to someone that can go answer me.
Todd: So that’s all built in, but this is a great way to talk actively about what’s going on.
Aakash: So it’s gonna become your artifact that you guys are communicating about the results. So let’s say at a weekly meeting and it’s basically automated for you.
Todd: Exactly. I mean, time to first use, how long does it take to get there? I already talked about path and journeys, so you can see that. So this is one stop shopping for everything you need on a given feature area. So that’s one start to how we run our business with Pendo.
I think the other thing is how we released our own agents and for that we have what we’ve talked about is agent mode and you can sort of see just a new way of working within Pendo, it’s not necessarily a PM agent, we have previous chats, things like that built in, but you can very easily say, hey, what can I do? And it’ll go in and it’ll start working, and it’ll essentially answer what it can do, and then you can see here it’s taken absolutely a platform approach, uses insights, adoption engagement, customer discovery, guidance survey performance, replays, debugging, so it’s the breadth of the platform, it’s everything from qualitative analysis to quantitative analysis.
So I can ask certain questions around hey can you compare two segments, let’s compare see gold customers versus silver customers and that’s what we’ll do. I mean, it’ll just go in and start doing the analysis on comparing these two segments. Now look, I mean, it’s taken a few minutes, it’s gonna do a little bit of work, but I think, oh, now it’s gonna ask me a question, which it needs to do if it’s some analytics based thing versus just hallucinating. So that’s probably a guardrail you guys built into the product.
Aakash: Yeah, look, it’s actually smart. It’s like, what do you want to compare? Do you want to compare how they’re using guides, you want to compare against their overall engagement, their qualitative data, you kind of need to do it. Let’s look at overall activities, say, visitors, yeah, let’s say over unique visitors per account.
Todd: And I think this goes to the point you were saying of you don’t want to just paste ChatGPT into your product. You actually, you sculpted the underlying LLMs to create a certain experience that matches the user expectations out of your platform, which are that it’s a very high trust analytics, user insights platform, so we can’t just get things wrong.
Aakash: Yeah, exactly, and now I can go in and I can show you using Pendo via Claude in our MCP server, and that is totally ungated, and that’s Claude using Pendo’s APIs to try to answer some of these questions intelligently and it’ll actually get to a lot of the same answers, it’s just gonna take a lot longer because it’s actually pulling raw data, it’s running Python over it, it’s doing a lot of really interesting work, but I think that’s one of the areas where it’s pretty exciting actually.
But yeah, and I am wildly impatient, so I’m gonna move over to another tab just to show you. I think this is kind of fun to show because this is one of the areas we’ve invested a lot in, but one of the biggest pain points that people have is there’s so much interesting product insights spread across your enterprise and how do I make sure I synthesize those and bring those back to the ones that really matter. So let’s say I want to look at top feature requests, and I click this button, let’s remove the date range and let’s go at it, it’s going to go and pull data from polls I’ve run, Gong calls, support tickets, really all over the enterprise and it’s going to surface.
And I think about one of the most painful parts of a PM job is just sifting through qualitative information. I mean, I’d often, in my past, tell PMs that one of their core competencies is going and sitting next to support to figure out what’s going on. And here now, if I click this, you can see we have this feature, you can see what the source is, I can see information around it, it’s a really powerful way to sort of go through all these sources and here’s actually one of more sources. You can see CSVs, Gongs, internal forms, guides, portals, Salesforce, found 152 responses and it organized for me really nicely.
From there, I can automatically create and link ideas, basically, I can push it right to Jira. So, the vision is, I don’t have to read a million pieces of qualitative survey in Salesforce or in Gong or etc., I can pull it right back here and drive decision making. So it’s another great example of AI and how some of these AI solutions are really making product managers’ lives easier.
Aakash: This is hands down one of my favorite applications of AI because something AI is really good at, especially if a tool like Pendo is purpose built for it where it’s not hallucinating or anything like that. There’s so many sources of information we’re getting, whether it’s Gong, Zendesk, social media, Slack. If you can get AI to help you analyze those and see the needles in a haystack, you can find so many good ideas.
Todd: Yeah, well, that’s a quick snapshot of what we’re doing, but yeah, it’s fun.
Stakeholder and Board Management (1:16:23)
Aakash: Amazing. So this is the roadmap, folks. Check out Pendo’s features if you want to learn more about how they’ve built them, and you can see now that you’ve heard from Todd how he thought about what is the vision, what are the safety ethics and guardrails, what is the strategy, what is the roadmap, how they actually then implemented those features. This is gonna be a really good exercise for you in building your product thinking muscles, specifically your AI product sense muscle. Which I think is different from product sense. AI, it’s non-deterministic, you have to think about safety, all these different elements, you have to think about cost, which we showed you guys. So there’s so many new elements that you need to develop.
When it comes to stakeholder management, this is a core product skill. Everybody knows about it, but the funny thing, and the reason I have it here is because as you become a product leader, managing your investor and your board level requests matters, and you know this better than anybody, because you’ve been managing your board for so many years. How do people think about that? Because boards are demanding more and more AI features, but you don’t want to just run after every shiny AI object.
Todd: Yeah, look, I, the board’s reacting to what you’re giving and what you’re presenting, so I encourage you to control a narrative. If you show up at a board meeting with no narrative, yeah, you’re gonna get crushed, why don’t you do this and why don’t you do this, have you thought about this? I think the key thing that I always focus on is I have a kind of first principle approaches to boards that these are really smart people, they’re here to help. We have vested interests, we’re aligned in the incentives, so everything they say is valuable.
So if they bring something up and why aren’t you doing this, or have you thought about this, my standard response is, well, one, if we haven’t researched it, let’s go take some time, we’ll come back to you in the next board meeting, or two, if we have, sharing what we know, it’s yeah, great point. We looked at that. Here’s what we found out, and we talked through those examples specifically. And look, maybe there’s something, some bias in how we tested it or some audience that was slightly different than what they anticipated, but we always try to share the why.
The other thing is, if you’re showing up your board just sort of looking for approval, you just want a stamp on your report card, all looks good, you’re using them wrong. They’re gonna work for you. They’re getting equity in the company for a reason. And so I always think about what do we want to bring the board that we want their feedback on, that we want to see what they’re seeing? Because here’s the other thing, a lot of them, what they’re doing is they sit on other boards or they’re involved in other companies.
So while we’re all in our company, it’s very easy for us to have groupthink and only think about our own world, but they’re seeing a whole another world out there. Now, it’s of course, couched in their bias, we have some early stage investors on our board which see all Series A companies and they ask me why I can’t build Pendo with 10 people and I’m like, it’s a little more complicated than that, folks. But then we have folks that are on later stage boards, which see slightly different things.
So your job is to come in and bring areas for conversation and for feedback and I think that’s how I treat it, but I think there’s a, I agree, this is one of the most critical skills in product, and this is where a lot of product leaders fall down, is that they have a hard time elevating to that next level around how do I communicate what success look like, how do I align what I’m doing to the business objectives that we have, understand where’s the business going and how does your piece roll up to that.
So critical. And that’s essentially what a board needs, how does this, how’s this ultimately driving growth for our business? How is this ultimately increasing shareholder value? That’s ultimately what the board needs to hear about. And so I think a lot just in general about how each one of our bets drive shareholder value. So that way I’m thinking the same thing they’re thinking.
That’s the other feedback I often will tell people is, no board I’ve ever been part of or I’ve wanted to be on, do I wanna see something just made for me? Although most people make things just for the board, but I think the lesson in this is thinking very deeply of how do you manage your business and what do you want to see. Why aren’t you looking at this more often? Why are you only creating it for a board? When someone comes to meet with me and it’s I created this just for you, it’s how else are you managing your business? If you just create this for me, what were you looking at? I wanna see what you’re looking at. Maybe it is smarter than what I wanted to see, but if it’s not, we should all be talking about why we’re not looking at the same stuff, right?
And that way, so I think about stakeholder management, I’m thinking of to be great at running your business, what do you need to be looking at? And I want to see the same stuff you do. And maybe you synthesize it, maybe you summarize a tab, but I want to make sure what I’m testing is, are you running your business well? And that’s what the board’s testing of me. Am I running my business well? Am I looking at everything? Am I focused on the right things. That’s ultimately the measure.
Aakash: That was a mini master class in board management.
AI Ethics and Safety (1:20:47)
Aakash: So we move up to the final layer, which I think you need to understand, and this is probably most important for people at your level, CPO level, the higher level people. You need to manage the overall AI roadmap to make sure that you have the right direction given down to teams about your position on data privacy, ethics, safety. What do people need to learn here?
Todd: Yeah, I mean, well, this goes back to kind of again, I use the term first principles like what do we do with data, what data is secure, what’s privacy. One is your company priority has a privacy and compliance position and stance. You need to make sure that whatever you’re doing in AI respects it and abides by it. You can’t do something different and crazy. I think understanding bias and fairness, I think is really useful and interesting.
If I ask a non-related question to my AI, is it fully ungated? One test we like to do in our AI is we’ll ask a question about who won some Olympics game, and it should say, sorry, I can’t help you with that, right? Because I try not to ask the really nefarious questions around very sensitive topics, but you could also, what if just by mistake, product manager released something that didn’t have any guardrails.
And you had some, you know, ChatGPT was in the paper this weekend around some kid that was talking to it about suicide and he gave it suggestions, right? I don’t know if you saw that article, but terrible article. And look, we should expect some things with technology. I’m sure there’s been technology about Reddit threads about bad things happening and no one’s, yeah, well, we, I don’t want to talk about whether we have moderation and things like that, but that’s a whole another conference like that’s a loaded term. So I’m just gonna parking lot that, but you need to understand these are all things that can happen and how do you deal with it and how does it work in different countries.
If you are working with companies in Germany or even Austria and they have workers’ councils, what is allowed amongst workers’ councils because you may come up with some amazing feature that people like the entire largest economy in Europe may not be able to use. Because they have these things that nowhere else really in the world has them and they’re, they have a lot of power. Similarly, are you working with financial services, are you working with healthcare? HIPAA, I think these are all things that if you’re in those markets, you have to understand.
Do they have a special posture towards AI because, and this is a tough one, because you’re seeing a lot of conversation around how much AI will be regulated, and right now I think with the US, our current administration, I doubt there’ll be a lot of regulation based on what I currently see, but administrations change. And the thing I always worry about is sometimes pendulum swing too far, so we may be in a, that’s one argument some folks are making for just a little bit of regulation now. So when perhaps someone else comes in power, they don’t completely over-regulate.
So I think somewhere maybe in the middle is the right answer. I don’t know. I’m not a politician, but I’m saying that these are all the considerations you have to have. So and here’s, if you build it in a way where you can’t change how it works, you are screwed. You may have to take the product off the market. So we build a lot of toggles, switches, all the AI we’re building, we’re building in a way that we can adapt it fast, on a dime. If it’s hardwired in, you can’t turn things off for different customers in different regions, and different locales, and different industries, you’re screwed. So when you’re building it, I would encourage everyone to build things treating it like quicksand because it kind of is.
Conclusion (1:22:03)
Aakash: Amazing. So, we have walked people through a roadmap of how to upskill themselves. As we said at the beginning, you can’t just slap an AI PM label on yourself. You need to go learn these things, but then more importantly, you need to go ship these in production at scale successfully. You walked us through some examples of how you guys have done this and what it looks like to ship successful AI features. This is the complete roadmap to become an AI PM and to ship AI at scale.
Use the toolkit that we’ve given you in this episode. Go out there, analyze episodes, go try out Pendo’s features, come back, comment below what you learned. And Todd, thank you so much for delivering this master class.
Todd: Thank you, Aakash. It was a lot of fun.
Aakash: So if you want to learn more about how to shift to this way of working, check out our full conversation on Apple or Spotify podcasts. And if you want the actual documents that we showed, the tools and frameworks and public links, be sure to check out my newsletter post with all of the details.
Finally, thank you so much for watching. It would really mean a lot if you could make sure you are subscribed on YouTube, following on Apple or Spotify podcasts, and leave us a review on those platforms. That really helps grow the podcast and support our work so that we can do bigger and better productions. I’ll see you in the next one.
