Check out the conversation on Apple, Spotify and YouTube.
Why most AI content misses the leadership tier (01:57)
Aakash: I’ve been looking online at all of the content on AI for PMs and I noticed two really big gaps. The first is that most of the content is written for IC PM tasks. How to write a PRD, how to conduct analysis on a feature. What if you elevate that from PRD to all hands presentation? What if you elevate that from analysis of a feature to metrics retrospective that you want to give to other C-suite leaders? That’s something all PMs have to deal with and especially product leaders have to deal with. So I brought in somebody who’s willing to share the real stuff, not canned hypothetical conversations, but the actual documents that he presented to his peers. Matthew Wensing has been gracious enough to create unparalleled insight into how a real VP of product at a real hyper growth AI company uses AI. So if you stay till the end of this episode, you’ll get to learn three things. Number one, how he builds all hands in just a single morning. Number two, how he runs metrics retrospectives with his peers. And number three, his entire weekly AI stack. So without any further ado, Matthew, welcome to the podcast.
Matt: Thanks so much for having me.
Aakash: It’s my pleasure. What are people going to learn today?
Matt: So today I want to draw back the curtain a little bit here and show you what it’s like for me to be adopting AI at the leadership level here at Customer.io. My job is no longer, although it was for many years, to be an engineer. As a full stack developer but always product minded, it is now to really help entire teams understand what it is we’re working on and why and to be that leader. I think that Claude can also help us, but it can also be very finicky, tricky, and hard to manage at times. I’d love to share how I’m learning to manage it well.
The all-hands presentation story (04:06)
Aakash: One of the stories you told me is that you had an all hands presentation to give to the company at 11am. Like most leaders, your week was booked, so you decided to wake up early at 5am the day of, and you built the entire presentation. Can you walk us through that and how you did it?
Matt: Yeah, happy to do that. The all hands presentation was broader than just me but I had a significant portion of it. Let’s say I was roughly a third of the presentation and the idea was to present our Q2 roadmap, which is a common thing that you need to do as a product leader, explain to the company what we’re working on next. I had a vision for what I wanted to do, but I came into it early in the morning and I am a morning person. So real quick aside, know yourself. In this case, 5am for me is like prime time. Not for everyone, but that’s what I did. I sat down and I had these Google slides and this template. I knew what I wanted to do and what I wanted to accomplish, but I really needed Claude to help me.
Aakash: So how did you work through it? What were the parts that Claude was able to do well? I know I was actually just recently giving a presentation and I had been editing with Claude till the last minute and then while I was giving the presentation I found mistakes which is like the worst case scenario. So how do you actually wrangle Claude so that you have the confidence that you can use it as a slide editing partner?
Matt: The first joke is that whatever you do, don’t click enhance this slide. We don’t know what that does, but we don’t click that around here. It’s a good running joke that I have that I terrorize people saying, I clicked enhance this slide for you. Don’t do that. What I did instead is I think you need to understand who are you communicating with. So go back to basics first. Who am I communicating with and what do I want them to take away from this? Reminding yourself of that is your anchor. In this case, it was the entire company, which is a blend of go to market, sales, marketing folks, and engineering. That got my gears turning around, I’m trying to provide a window into our product roadmap and our plans for Q2 to folks in the company that are not involved in actually the making of this material. That started my gears turning around what’s the raw material that I want to put into Claude. If you think about AI as excellent at taking raw materials and turning them into something else, the first thing you want to do is just have that inventory of raw materials and go through that in your own mind. Don’t jump into building the presentation. It’s like, let’s take an inventory of what we have on hand first.
Take inventory before you open Claude (06:19)
Aakash: How did you go from there?
Matt: In this case, the raw materials I had were fortunately prior materials that our company already had. One of those that was really valuable was our demo day presentations. I’m showing you a screenshot now. This is the Zoom recording of the demo day. This is me going into Slack at 5:10am and going, okay, there’s a link to the Zoom recording of the demo day. That’s excellent raw material for this presentation. But this demo day was for the engineering team to share amongst themselves. So I was like, okay, this is great raw material but it isn’t shaped correctly. It isn’t actually pointed at the right audience. This is sort of engineering for engineering’s sake. I had this and I downloaded the eight files from this. I said, give me the video and give me the transcript with all the timestamps. That was the first bit of input that I put into Claude.
Aakash: Is there anything around using Zoom transcripts people should be aware of? Can you just copy paste a whole transcript in?
Matt: Well, I did try that at first. I copy pasted the entire transcript in and I gave it very clear directions in terms of what I wanted to do. I gave it framing, I gave it context. I treated it like a junior employee, which is something you’ll hear me say elsewhere as I walk through these examples today. Here’s the thing. All I want you to do is extract the timestamps from this because what I really need is this for my screenshots. I want to put screenshots into this presentation. I need these screenshots. The only place I’m going to find them is in this Zoom call, but I don’t have 50 minutes to go through the Zoom recording. So here’s all the timestamps. I need you to help me find the right places in this video to grab these screenshots for me.
Aakash: So if we walk through this deck, which parts of this deck are you and which parts are Claude?
Matt: The templates, the shapes and the art, are obviously from our marketing team or our brand design team. That was handed to me. But the parts that were me were the shape of the narrative. I’m a huge believer in storytelling and upping your craft in terms of storytelling. I said, okay, we just had a huge launch. This was the middle of April. It’s important to acknowledge, when it comes to stories, people want to know where do we leave our hero, where do we leave off last? In that sense, it was we need to acknowledge the launch, acknowledge what we’re doing immediately post launch, and then use that as a bridge into the rest of the material. I decided ahead of time that’s just my own storytelling craft, we’re going to do a few slides on adoption metrics, where we’re at, and what are the fast follows that we have, and then we’re going to go into the strategic. I wouldn’t outsource that to Claude. Perhaps you can, and I could turn that into a skill, but when you’re doing these things maybe three times a year, it’s not worth the tooling investment just yet.
Matrix multiplication, pivoting content into strategic shape (12:14)
Aakash: How did you go from there?
Matt: The raw ingredients for the strategic look ahead part were the Zoom transcript and the strategy doc. It needs to know the three themes of our strategy. What I did with those two and this is where Claude created a ton of value, we had all of these presentations, these lightning round presentations of what we’re working on next. I also had the strategy doc. In order to take that Zoom call and turn it into something more strategic, I basically needed to pivot that Zoom call into the same shape as our strategy docs. I said, I’ve got this doc which outlines our three themes for the year in terms of product and engineering. Can you please go through that Zoom call recording now for me and instantly organize all of those presentations by the category, the investment theme, that we have for the year. That was instantaneous.
It was like I have this raw ingredient over here, I have this one over here, and I essentially want you to do what I call matrix multiplication. It’s a transformation. It’s like a pivot. I want you to pivot this content to match this content. That becomes something I can actually use and go, okay, this content is the shape I need. This is the strategic shape, but this is the raw material over here. Can you please adapt this to this? Once I did that, it was like you almost feel that audible click of, okay, now that Zoom call recording is strategically shaped and I can start to flow it into these slides.
Aakash: So that happens pretty quickly. I imagine you could have done most of what you just described in 30, 45 minutes. What went on from there? How did you edit it and polish it?
Build slides first, talk track last (13:50)
Matt: The edit and polish from there was the talk track. The talk track was just as much work. This talk track was really important to nail the beats of the story. I had it do that last. I believe that some people write the talk tracks and then they do the slides. In this case, I chose to do the slides because I knew what I wanted people to see. I’m a big believer in show, not tell. So I want to get the show part right first. But then the tell part becomes and this is what I did, I actually took screenshots of the finished slides and then I fed those back to the same Claude session. I said, hey, write the talk track for this. And what I don’t want you to do is I don’t want you just to regurgitate. You have all this context now. You’ve ingested the entire call recording. You’ve ingested the entire strategy docs. You know what I’m showing. I’m showing you what I’m showing. Now I want you to write the talk track last that doesn’t just repeat what’s on the slide, using all of that context that you have. It came up with a much more interesting talk track as opposed to write my talk track and then I’m going to put together some slides. The order in which you do things really matters.
The eager junior problem and how Claude races ahead (15:08)
Aakash: You said something that stuck with me on our call. You said that Claude sometimes has a leash and sometimes it goes too far and you have to reel it back in. Can you explain that and give me a time you had to reel it back in?
Matt: Reeling Claude in is a constant challenge. I have a rule when it comes to and I even published this in my how to work with me doc that I share with new hires, that a lot of junior employees, which I would consider Claude one of, very talented but very junior, is very eager to please. Because of that eagerness, it will go too far too fast. This one was an example where I was doing a write up like a root cause analysis on some metrics of ours. I had extracted some data, I gave it to Claude and asked it to look at that data and I was working through it. The first thing it did was want to go through and just generate a whole pricing strategy doc for me in Word. Because it’s super eager to please. I think a junior employee will hear a prompt or a first instruction and go, I know what to do. I want to please my manager. I want to please the boss and I’m going to go do this thing and just knock it out of the park. They won’t ask any follow-up questions. They’ll just race to the finish line and go do that for you. And then when they come back, inevitably you need to change it substantially because they didn’t ask enough clarifying questions first.
I don’t know what it is in the training. I think there’s likely a way that you can improve your harness and get it to behave differently. I’ve been iterating on this with my own, but helping it not rush ahead. If I share a strategy document as a Word doc, people are going to think I’m crazy. That’s not how we work. But it’s so proud of itself. It just launches into that. So I have a habit of doing what I call very iteratively rolling context into the session so that it doesn’t have that eager to please. If I see that eager to please, hey, do you want me to do this next? I very quickly say stop. Stop recommending the next step. I will tell you when I want you to do the next thing. Just to kill that thinking out of the session. It also is very much like drip torture. At some point it keeps nudging you like, hey do you want me to write the thing, do you want me to write the thing? And it almost feels like you’re being coerced into saying yes. You’re like, it’s not time yet. You don’t have enough context yet and you don’t know that. You really need to stop asking me to run ahead and generate the deliverable.
These sessions, they can take, it’s better to have that 50, 100, even 200 iterations session with a great deliverable at the end than to say yes to that first ask to generate that deliverable and then try to revise it. Because inevitably that first draft is just full of so much slop. I actually have a way of thinking about slop. I think of them as micro hallucinations. It’s not that it’s totally wrong. It’s like, okay, you need a doc, but here’s a Word doc. Well, wait a minute. You’re not hallucinating in the fabricating data sense, but you’re hallucinating about the way work gets done here. We don’t use Word docs. It’s trying so hard to please that you end up having to just tear down and redo so much work. It’s not worth it.
I’ve developed a few tricks on how to force it to kind of slowly crawl along with you through the mud and really earn its right to generate that final product instead of rushing ahead and developing it right away. We think of that as like the magic of AI, it can just do this thing so quickly. But as a leader, you don’t want to microwave your output. If that’s all you’re doing, that’s low value. You want to really slow cook these things more often and produce something that is really compelling, impactful, resonates with your audience. You’ve got to slow it down enough to build that story that you’re trying to tell.
Aakash: So how do you slow it down correctly? How do you avoid kind of becoming the slop cannon?
The biology metaphor session begins (19:02)
Matt: I have another example here, and this is the most extreme version of that. I do a lot of my interactions with AI by voice, especially when I’m on walks. I tend to use the mobile app more often for this because I love the voice mode. In this case, I was using Opus 4.7 and I started to use the microphone button and I did something very deliberately for the first time having learned this enough. Like a lot of leaders, I do tend to use analogies a lot. Analogies are super helpful to clear away the noise. The analogy’s purpose is to say, this is the main idea embedded in this analogy. We’re going to transfer that over later to the domain we’re talking about, but let’s actually just work through the analogy.
In this case, I was trying to build an entire lifecycle model. I was thinking through the lifecycle of our customers and how they kind of enter the business, sometimes they churn, they expand, all these things they do. But I didn’t want AI to know that yet. So what I did is I insisted on using a biology metaphor. I just started talking in terms of, okay, let’s imagine we have this ecosystem. It’s like the water cycle or a lifecycle. Everybody knows that caterpillar drawing. Let’s think in metaphors for a minute.
What’s interesting is that Claude at this point has all this context on me. It knows I’m a VP of product, it knows that I work at Customer.io, it knows things about my personal life frankly if I share the things I’m doing outside of work. So it’s putting together this picture of me and it’s always trying to add value by anticipating or reading me or understanding, okay, I know where he’s going with this. And then it jumps to that next step.
What was fun about this exercise is, start here, assume you have a 2×2 matrix and you have each of those representing a stage of life where things begin at the bottom left and proceed to the top left or the bottom right, and then they finish at the top right. I’m talking in very abstract terms. At this point, I think it has no idea where I’m going with this. And that is to me a virtue, a benefit. Because if you look down at how it responds, it goes, okay, before I give you a number, I want to check a couple things here because the count swings a lot depending on what you mean. I asked it, hey, can you summarize the number of permutations of these pathways or these lifecycle things that are going on in this fictitious system? It is like so far out there now in terms of its brain. It’s like, okay, this is weird, I don’t know where he’s going with this, but I’m gonna go on this journey with him. And you can notice at the end of this response, Claude doesn’t know where we’re going yet. What is it doing? It’s asking a clarifying question. And I thought that was such good immediate validation that what I was doing was working. Rather than jump into, do you want me to generate a presentation on like customer lifecycle and bring in all of it, it didn’t do that. What it did instead was, hey, before I answer this question with this very esoteric thing we’ve never talked about before and I don’t really understand where you’re going with this, let me ask a clarifying question. A senior person asks clarifying questions of a leader before they go do a thing. To me this was a sign that it was thinking more like a senior person.
The game night rule for layering complexity (23:21)
Matt: You could probably codify this into a skill or a prompt, where you say, please ask clarifying questions before you act. But this was an example of me going, okay, even if I don’t do that, can I lead it down a path? Here we’re over and over again going through these layers. What I also did was I started to add complexity. At the very beginning of this, this is a very MBA-like 2×2, something extremely simple. By the middle of this, I’m starting to add in, okay, the rules of the game say that actually the thing can enter from the bottom right or the top left or the bottom left. And it goes, oh, I get it.
Have you ever been to one of those game nights with friends and somebody explains the rules of the game? There are two kinds of people. One person explains the rules and they feel like they have to explain every single rule but also every exception to the rule as they go through. In my experience, people’s eyes tend to glaze over because they’re just like, okay, so there’s these rules, but then there’s all these exceptions to those rules. So there’s rules on rules and they’re so lost in terms of understanding the overall game. What are the goals? What are the objectives? How do you win?
With Claude it’s very similar. It benefits when you start very simple with here are the rules of the game, here’s how we’re going to win. But I’m not going to give you all of the exceptions to the rules and the nuances at first. I’m going to give you just the very crude heuristics or very crude outline of what I’m thinking. Then like I said, iteratively layering in the complexity. Each time I added in that complexity, I helped stabilize that foundation of we’re not relitigating or revising what we’ve already established. The rules of the game, these are the basics, but we can add in this complexity.
If you dump all of this complexity in at once, it’s just like a person. It gets indigestion, it gets mental fatigue and it goes through and it just overdoes it. You end up with this super complex, lost in the woods experience where it’s really proud of itself and maybe you should be too, but you’re not because you’re like, where did I go wrong here?
I think you’ve not only done the AI a disservice, as a leader you’ve also made it a little counterproductive to yourself. You benefit from asking, am I really developing clear thinking each time I go through these iterations or am I rushing to conclusions? Another thing that can happen is if you add in too much complexity too fast, you yourself aren’t challenging yourself enough. A lot of this was me going, oh, that’s interesting, yeah, I guess this is another aspect of this that I hadn’t really thought about. I was giving myself time to think instead of again rushing to that conclusion.
Revealing the domain only when the model is clean (26:08)
Matt: At the end of this, what did I need? I’m actually trying to build a pretty holistic view of our customer base. It wasn’t until many iterations through that I finally told it what this is for, what’s the purpose of this. And then it was really funny. As soon as I told it the purpose, it got really excited and it started running down the path. There’s a response I shared where I go, do you want me to draft this as a Notion page in a pricing philosophy section, or is it a one pager that Colin could have? Colin is our CEO who I report to. No, let’s stay abstract for now. And it says, I was sort of afraid of telling you what this was for because you would get a little excited like a junior intern. We need to stay academic for just a little bit longer. This is me literally coaching it through. I was afraid this would happen, but I held things at bay for as long as I could. I wrung all the value I could out of the abstraction and out of the analogy. Only once I was sure, okay, now I have to tell them what this is for, then I did. But once I did it was like, I get it. Now we’re going to take the output of this exercise and pivot it or apply it to this domain.
I could have gotten to the end of this exercise and honestly applied it to any number of businesses. I could apply this to anything from an e-commerce business to a bakery to a dry cleaning business to Customer.io. The fact that I didn’t let it assume that it was relevant to Customer.io at the beginning let me build what I think is a much more clean mental model to work with, and then apply that to a domain. Almost as a stress test. If this generates nonsense when applied to Customer.io, then I know something’s wrong with the model. But if it generates things that line up with reality, it actually gives me more confidence that the model itself is correct.
Aakash: I think there’s a really important point embedded within here. As a leader, we don’t exist just to generate Claude outputs. We exist to create mental models, to reframe, to drive alignment, in this case with a bunch of other leaders on the pricing philosophy. What we need to do is actually present a way for them to think about these pricing options. So you used Claude as a thinking partner to drive what you’re going to show in this metrics retrospective. You didn’t just jump to Claude, let’s build out this metrics retrospective. You actually focused on the thinking.
Matt: Yeah. Focus on the thinking and then use what I think it’s really good at. When I remember when GPT-4 came out, I remember telling my father-in-law who is a doctor and does a lot of intellectual work, and I remember saying, it’s getting scary good. What I realized was it was suddenly able to take what I think is the bulk of a lot of what we do as leaders and as knowledge workers. I intentionally call it blue-collar knowledge work, and people are like, what do you mean blue collar? Blue-collar knowledge work is taking information and pivoting it or translating it from one place to another, this is a slide deck, it needs to be a Google doc, this is a Google doc, it needs to be a slide deck. For a long time before LLMs, people were paid to do that translation work or that transformation work. That is the sort of lowest tier of work that’s been wiped out. If you need to transform the form factor of this material, that’s not enough anymore. I have to bring some kind of novel way of thinking to the table.
It’s not the thing on the right or the thing on the left. Those are the source and the target. The source and the target are not new. Even the work to translate source to target is what we used to do. That’s where a lot of middle management and consultants would work away. The real work that’s left for us is how do I choose the right source and how do I choose the best target? If you think about the targets as the stories you tell, the shapes of those stories, the form factors, the deliverable, should this be slides, should this be this, should this be that? That’s a choice. You’re still making that choice. And in terms of the sources, is the source a clean mental model about pricing philosophy, is the source seven conversations in Slack? My value is actually choosing the sources or set of sources to use and then being very deliberate about what I translate those into. I’m not just running the function. I am being very deliberate about the inputs and the shape of the output.
I’ll give you another example. In this exercise, I had developed some cute terms for some of the things I was working with, names for each box, like unicorn or horse. And Claude just ran with that right away. It just launched into that, putting those terms into the target. I had to come back and say, hey, here’s what I ended up creating. It ends up in Notion. What’s different about this compared to what you had generated? And it confessed to me that it said, I used those terms immediately. I didn’t stop and think, those are new terms. If I put those into the story, if I let those get into the target output, there are going to be people who read this who are like, what the heck is a unicorn, what are these new terms? They’re going to focus way too much on those and resist. There’s always a chance that somebody goes, those are silly or I don’t understand those or that’s jargon. It didn’t know that. As a leader, I had to tell it, don’t use those terms in this output because the audience isn’t going to react positively to those terms yet. We need to introduce those later. Maybe a month from now. It doesn’t have that social IQ. That’s not built in.
As leaders, we need to bring that to the table and go, how is this story going to resonate? Are there things that people will focus on? Claude just doesn’t know that yet.
Aakash: Would love to see it. Let’s see how that exchange went.
Matt: This was me coming back to the LLM afterwards and saying, hey, so I did just share the final document with Jason, who’s our CMO, and a few director reports. I said, I’m curious why an LLM like you might struggle to write this for me automatically. In other words, I wrote this instead. It goes, the voice problem, it doesn’t know how to speak like I speak just yet. I think we can work on that. This one I’m less optimistic about, at least for now, the political calibration. So you knew to drop the animal names I had in the source material and use simple small and large sophisticated instead. The choices of those were deliberate. You know how to read the room. I don’t mean political in the business-as-politics negative way. I mean more in the sense that people are political animals. We can’t escape the fact that we’re all judging and evaluating what we read as we read it. It’s second nature. We do that to help filter out the noise. It’s just not good yet at guessing what’s going to translate well for your audience.
As a leader, being on guard or vigilant about those, that’s not even a micro hallucination. That is a misreading of the room and being too eager. That’s the same thing a junior employee would do. They wouldn’t realize there’s a lot of baggage around that term. It reads some source material from three years ago and adopts a term, and you should know as a leader there’s a lot of baggage and maybe even some careers attached to that term. Claude just doesn’t have that historical context yet.
And then the persuasiveness. There’s a reason the best authors and writers and storytellers of our generation are still not just using AI to generate those stories. It’s bringing the reader along with you, knowing that they just came out of an all hands, knowing that they just had the biggest launch day in their history, knowing where they come from and leading them through. That’s another thing you can’t delegate yet. You need to be really good at going, what’s the mental head space or emotional space of my reader as they pick up this document on a Tuesday? Are they really excited? Are they exhausted? Did they just get into a board meeting room after traveling all night? You need to think about those things. Claude just isn’t going to yet. But that’s where I think you can be exceptional as opposed to just accepting what AI generates for you.
How to decompose problems before building anything (36:26)
Aakash: So we just walked through two of the most important examples that a product leader needs to be able to use AI for, all hands style presentations, metrics retrospectives. If you were to synthesize against those, what are the key lessons for how to use AI and how not to use it for these types of work?
Matt: The shortest version, which I think cuts across all of these, is that AI for leaders is ultimately a test of how good are you at decomposing problems. AI is very good at solving a problem, but it will simplify the problem space if you don’t properly decompose it. As a leader, not one-shotting something means not just iterating with it, but being very deliberate about decomposing a nasty problem into its pieces and then saying, okay, the right series of transformations, starting with this if we want to get here, how do we decompose this problem into a series of transformations that I’m confident you’re going to be good at performing? Then we’re going to get to where we need to get to.
Where we fail is when we flatten problem spaces and oversimplify. Then it’s just, well, clearly this solution is this. Challenge yourself to really take a nasty problem or a deep problem in your business and really explode it, decompose it into all of the pieces you can, and then put those pure observations, those pieces, into the context window before you start to then assemble a solution. When you oversimplify and you just have this sort of flat projection of the problem, that’s where you get slop. The people reading it go, this thing doesn’t really understand the multi-dimensional nature of this problem, the complexity of this problem, why we haven’t been able to solve this problem yet. Forcing yourself to dwell on the problem for long enough to really decompose it and see all the pieces separately is where you’re going to create the most value.
Why AI alignment decks backfire on executives (38:17)
Aakash: I think this relates to a central thesis we have, and you have a take on this that I don’t think other people have said anywhere else. What happens when leaders try to use AI to drive executive alignment?
Matt: I think we know what happens. I think that’s bound to fail. People can feign alignment really easily. I also think it’s important to define alignment, what do you actually want? Some cultures have a disagree but commit attitude. Others have a we really need to agree on all the details, like a shared consciousness version of alignment. If you use a degenerate alignment, I think executives are the best at filtering out noise and detecting slop. Depending on where you are in that hierarchy, you’re going to get a variety of responses to what you’ve created.
If you’re more senior and you do that, you’re going to have a lot of people who feel obligated to smile and nod or go along with the flow or accept what you’ve created and say, okay, I guess we can work with this, but have you considered this? If you start to hear those things, you might have a problem. That’s if they feel psychologically safe to even say that. If they don’t feel safe, they’re just going to roll their eyes or ignore you, which is the worst kind of misalignment.
If you’re lower, director level, senior director, higher up VP, senior VP, not C-level but elsewhere in that leadership hierarchy, you’re going to find out that people ignore your work or ignore your output and don’t even feel an onus or a responsibility to take it into account because they’ve basically filtered it out as noise. That’s really disheartening. You think you worked hard on something and then it doesn’t get airtime or it doesn’t get incorporated into the corporate conversation. You kind of know when you’re being ignored because you can see that you’re not getting attention. But the diagnosis might be, yeah, you are generating a lot of your points of view in that very flat way. The best leaders, the ones that are going to help you grow in your careers, are going to be the best ones at filtering out that stuff immediately.
The symptom of that AI slop generation or alignment goal is going to be different depending on where you sit in that hierarchy. But the ultimate outcome is going to be that alignment doesn’t end up happening.
Matt’s full weekly AI stack (40:56)
Aakash: So that really puts a nice summary over what we’ve just described across these two really important leadership tasks. Now let’s zoom out and I want to understand what is your overall weekly stack with AI. What’s always on? What are you using it for? What’s the whole list that people should be and shouldn’t be using?
Matt: The reality is as a leader I live in a few places. One is yes, I live in Claude desktop in this case, and things with Cowork and Claude in general as I just walked through some examples. The other place I live all the time is Slack obviously, and Customer.io has done a really great job of bringing more and more AI and automation into Slack. We have a growing number of agents that are internal and I’m using those on the regular to do a few things.
One of them is ad hoc analysis. We have a bot and you can see I have a 215-reply thread going with it. The idea here was I had 2,000 customer records. I want to do some analysis. I fed it to this. It has access to Snowflake and it’s doing some querying for me where I can just use natural language to do this analysis with it. This is my go-to for when I have a question and I need to verify some data that’s going into some research or report or summary for other executives. I’ll say it on two fronts. One is yes, have a bot like this, but the other is having a data team that can chime in and either help unblock it when it’s not performing the way it should, as I like to say, kind of kick the vending machine or slap it or verify the data. It’s always important to say, I’m not just going to take a non-deterministic answer for this.
Another way is Josh Childs, a member of our team, and many others have built their own tools to do really cool things. We’re a fully remote async company. We have over 400 employees now. That means there’s just a ton of conversations happening. So Josh built for the sake of the product team a scanner. It uses AI and it goes through a few dozen channels that we have and just tries to find any conversations happening anywhere where a product manager should probably be involved. We got to the point with our company where it was just too difficult to expect anyone to read through all the threads and conversations all the time.
This has been a huge help. It’s a scanner. We don’t think of this as a police car going around patrolling. It’s more of our radar or our sonar. It’s on all the time. It’s really helpful to just deep link to a thread where it’s like, hey, in this channel or on the support ticket with this customer over here, I see signs that a product person probably needs to weigh in, but I don’t see any product person yet. Then tuning that to report at certain times of day and not be overly zealous. As a leader, I’m able to take those and say, hey, I see this conversation happening. I see where a product person could create some value for the company. But let’s also think of this as a process improvement opportunity or something we want to work on later and then follow through and tag folks that can improve the way we work. It helps me stay close to the ground while I’m 200 replies deep in some analysis or with Claude going through some kind of abstract exercise. The fact that I have this scanner running all the time still helps me pay attention to the details.
Chiefys and how Customer.io audits strategy docs (45:06)
Matt: And then one more, and I think it ties into what we were talking about earlier. We call this Chiefy. This is something that Colin made and he’s mentioned this on stage before, so I feel comfortable sharing it. This is a bot that works again inside of Slack and it has two primary use cases. One is anytime we create something new, like that analysis, like that pricing documentation, we can run that through Chiefy. It has this corpus of the 20, 30, 50 relevant company docs that are kind of the gold standard or the ratified, verified documents that we operate on, like the operating model if you will. It can help find discrepancies. That can work two ways. The other use case is there’s a discrepancy because we released something new. We really like it, it’s aligned with our strategy, but hey, we have to go update all these other docs that have already been written or we need to correct those. As a leader, it’s really painful to publish something or create something that gets alignment this month, and then we do something new in three months and that document now is either stale or showing its age or needs to be updated.
It’s great to use AI to automatically go through the dozen or two dozen documents that you have and just help audit those to know, hey, we need to bring those up to date. Or hey, eight out of these other 12 disagree with this. You might not realize it because you’re very recency biased. Is that intentional or not intentional? Being that accountability check to say, oh, yeah, we didn’t mean to say that we’re going to do this instead of that, we’re actually doing both. Or, oh yeah, we’re actually changing our strategy a little bit. Let’s go back and change those documents. Super helpful. No one has time now to go back and look at all those. That tends to be why Notion gets stale because there’s just not enough time to keep up with the auditing and reviewing of past artifacts.
Aakash: This is really cool. So what do people need to do to reverse engineer your Slack setup with AI here?
Matt: It depends on how your company works. We took a strategy of, in a controlled way, letting there be experimentation with open Claude and other agents like it. We actually have our own version of open Claude that we’re working on where it’s an agentic loop. We have individual team members who have that predilection for being technical who are building their own instances, and then we’re supporting them as a company and saying, here’s how to host it, here’s how to make it secure, here’s where it can run and live. That enablement is really key, making sure that there’s a budget and that space, that margin for people to experiment. It starts from the top, heavy emphasis on experimentation and building with the latest and greatest. This all kicked off in February as open Claude was exploding. So reverse engineer it. Open Claude, your own version of that, having a place for people to host their own instances and then letting those have access to Slack. From a leadership team perspective, budget, support, and then lead by example. Use these tools, maintain these tools, give feedback on these tools, create some of your own. That’s all going to really help to drive that flywheel.
Aakash: I think that’s really eye opening because I keep talking to people about open Claude and they keep saying, well, my company doesn’t allow it. I like your approach of we’ll create our own version that is enterprise data safe so we can use it with our enterprise clients and then deploy it. So most content you’ve seen online, it’s teaching you some advanced Claude code setup that is loading in millions of context files and your entire Notion and your entire Slack and collecting all your MCP. We just showed you the realistic, simple version of how you use AI to do the most important tasks that a product leader has to do and how you even enable your teams with some more advanced use cases to use things like open Claude. If people want to get in touch with you to learn more, Matthew, where can they go?
Matt: You can message me on LinkedIn. I do see those and I’m always recruiting so you can find me there for sure. I check those. You can also find me on X at Matt Wensing.
Aakash: I think it would be an amazing PM job if I were a PM. So reach out to him. If you are one of those AI native, AI forward PMs who have watched all the way to the end of this episode, that means you are embracing AI in a way that I think Customer.io would appreciate. Matthew, thank you so much for actually showing the real stuff. Nobody shows the real stuff. Really appreciate you.
Matt: You’re very welcome. Thanks for having me.