Check out the conversation on Apple, Spotify and YouTube.
Intro (0:00)
Aakash: Understanding which surface to reach for which use case becomes one of the core PM skills that will help you become 10x more effective. Join Jyothi. She’s been an AI PM since before it was cool. She’s been an AI PM at Netflix, Meta, and Amazon. You posted this on LinkedIn and it caught my eye. You said that you won your internal hackathon against 30 engineering teams and you used this concept of adversarial agents. Anthropic had just released a blog post around harnesses and long running agents. So I looked into the blog post and they had this concept of adversarial agents. That was what got me the hackathon, right. Where does the product manager line end and developer line begin in 2026?
Jyothi: Get comfortable with building. Get comfortable with, say, Claude Code, with all the Claude ecosystem that we learned today, and get comfortable building and putting your ideas out there.
Aakash: How do you use Claude Design? How do you build a knowledge based MCP server for all of your PM context to make Claude 10x more productive? That is what we’re going to answer in today’s episode. Now there’s this new role coming up called AI builder. Anthropic’s adopted it. OpenAI’s adopted it.
Jyothi: Making and building is easy now. Taste is what is important for us to develop.
Aakash: Can you do the big reveal now and help us get that setup going in Claude Code?
Jyothi: So here’s the thing.
Aakash: Before we go any further, do me a favor and check that you are subscribed on YouTube and following on Apple and Spotify podcasts. And if you want to get access to amazing AI tools, check out my bundle, where if you become an annual subscriber to my newsletter, you get a full year free of the paid plans of Mobbin, Arize, Relay.app, Dovetail, Linear, Magic Patterns, DeepSky, Reforge Build, Descript, and Speechify. So be sure to check that out at bundle.aakashg.com. And now, into today’s episode.
So I’ve been thinking about something. We’ve had advanced tutorials on Claude Code, with analytics, on PM OS setup, but how do you actually take the entire Claude ecosystem and make the most out of it from scratch? I keep getting DMs from people who say, “This episode is too complex,” or, “I’m not at this level yet. I’m still stuck on ChatGPT.” If you’re one of those PMs, this episode is going to build you from 0 to 80. We can’t get you from 0 to 100 in a single podcast, but we’re going to get you the 80% you need to know in 20% of the time.
I have brought back Jyothi Nookula. You guys loved her last episode, and specifically the feedback I got was that her structured communication was amazing for beginners. So she’s going to break down for all the beginners how to make the most out of Claude today. Jyothi, welcome back to the podcast.
Jyothi: Super excited to be back. Thank you for having me.
How Jyothi won a hackathon against 30 engineering teams (2:47)
Aakash: Jyothi, you posted this on LinkedIn and it caught my eye. You said that you won your internal hackathon against 30 engineering teams and you used this concept of adversarial agents. Can you break down exactly how you won the hackathon?
Jyothi: Yes. So a few days before the hackathon, I was trying to see what I could build, and Anthropic had just released a blog post around harnesses and long running agents. So I looked into the blog post and they had this concept of adversarial agents, where you build an agent and then you set up configurations in another agent telling it what matters most to your company, in a way, not like an eval, but more around capabilities that you want your agent to test. And so I started with that idea and then I said, let me take this idea. I went into Claude Code and I was jamming with it for almost a day with different configurations, and there we go. I had an adversarial agent evaluator running. It was exactly how I pictured it to be. I even pointed it at our company code and integrated that into an actual production running code. And that was what got me and my team the hackathon prize.
Aakash: So that’s the promise for you guys. We are going to help you get to that level. Where do we start, Jyothi? How can we break this down in a structured way so that people can get to this level at the end of the episode?
Jyothi: Great. We’ll tackle it today. So we’ll start with understanding the Claude stack first, and then getting into some of the basics, like how do you use Cowork, and then getting into Claude Code itself. So let’s get started on the Claude stack. So at the bottom of the stack is your models. So Claude has Haiku, Sonnet, and Opus. They’re all different intelligence profiles, and which one you need to use when, they have different cost profiles, different intelligence profiles. So that’s a decision framework we’ll get to in a second. So this is your layer one.
On top of your models is what’s built, your surfaces, which use to access these models. So your surfaces could be like Claude.ai, which is on your browser. It could be a desktop app. It could be your mobile app. It could be your Chrome plug-in. These are all the interfaces that you use to interact with Claude. Now these are not the same product with just different UIs. They have completely different capabilities, and understanding which surface to reach for which use case becomes one of the core PM skills that will help you become 10x more effective. So this is our second layer.
Now on top of this layer is your knowledge base. Now this is where your institutional knowledge lives. Your projects, your skills, your memory, your custom instructions. Now this is the layer that I think most PMs underinvest in. It’s this layer that makes Claude go from being a generic chatbot to actually knowing your context.
On that stack is your layer four, which is your integration fabric. For example, your MCPs. Now MCP connects Claude to every external system that your organization uses, like your Slack, your Google Drive, your Jira, your Salesforce, your internal databases, or your own local files. Skills are what extends what your Claude knows how to do and what to do with that data. So this is your layer four. And now on top of that is your agents and orchestration. This is where your Claude Code, Cowork, Design channels all sit.
Aakash: Got it. That’s how I think about the Claude stack. What do people need to know about layers one and two in order to make the most out of the top layers?
The Claude stack overview (4:30)
Jyothi: Yeah. So let’s get into the models. Now, Haiku is your speed machine. It’s the fastest, cost efficient, and it’s really great for tasks where you need volume over depth. So let’s say you are trying to generate a large number of variants of something, or triaging like a pile of documents, or you’re doing some quick classification, or maybe even some tagging. Haiku handles this really well. Now the output won’t have the reasoning depth of like your Sonnet or your Opus, but for tasks where depth isn’t needed much, Haiku is sufficient for your use case there.
Sonnet is where 90% of my work lives. It has the best quality to cost ratio. So when I’m drafting PRDs, or I’m synthesizing user research, or I’m doing competitive analysis, or I’m doing stakeholder briefs, or I’m thinking about roadmap, I use Sonnet. Sonnet handles all of this extremely well.
Opus is for your high stakes, high complexity reasoning tasks. So let’s say if you’re doing some complex trade-off analysis, or you’re synthesizing genuinely contradictory research, or you’re doing some long horizon planning where you need the model to work through second and third order implications, Opus is really good. It has really strong reasoning capabilities. But I’ve also noticed from my day to day working with Opus that it also tends to get into this hallucinated stuck mode a little bit quicker than Sonnet, where I would use Opus and it would get into one reasoning decision point, and it would keep revolving in that local maxima, and I would have to literally turn off the chat and move to a new chat and then start all over again to get it out of that thinking mode, for example. And that’s when I sometimes move back to Sonnet, because even though it may not have as high a reasoning, it’s generally a very efficient model to work with, and it’s also more cost efficient than Opus.
Aakash: Got it. So bring the right model to the right task. It sounds like for 90% of the tasks for PMs, you’d recommend Sonnet.
Jyothi: Yeah, I think that’s a good place to start with, and then if Sonnet doesn’t work for the depth that you want, you can always open up a chat with Opus and start there.
Aakash: What do we need to know about the next layer?
Which model to use: Haiku vs Sonnet vs Opus (7:17)
Jyothi: So next is your surfaces. Now Claude.ai, which is your web or browser, I think this needs no introduction. Everyone’s pretty familiar with this. This is where you can use to chat with it. The downside is that it doesn’t have access to your local system. So if you have some files that you want to access, Claude.ai may not be able to directly go and change. Of course you can have an MCP server, but still I don’t prefer it for anything that needs local access.
That’s when I use desktop. So my Claude Cowork runs here. It’s able to access my files. It’s able to access all the other systems that I have, integrate, and run some scheduled runs, which I’ll show you in a second. I built a podcast guest prep agent in Hyper Agent. The job is simple. Before every interview, give me the guest’s recent appearances, strongest arguments, company context, sharp question angles, and stuff I should avoid asking because everyone else has already asked. For this run, I pointed it at Howie Liu, CEO and founder of Airtable. The useful part is it can actually go do the research. It’s browsed, pulled sources, worked across files and integrations, and then turned the whole thing into a brief I can use before I hit record. Here’s the output. Recent appearances, public arguments, company context, question angles, and what not to ask. This is the kind of prep doc I actually want, not a generic summary. It shows what they believe, where their thinking has changed, which questions are obvious, and where the thinking tension might be. Then I saved it as an agent. The output is useful, but the saved agent is the real goal. I don’t have to rebuild the whole thing. I point the same agent at a new name, and it already knows the format I like, the sections I care about, and the kind of question framings I come back to. Podcast prep is one example. The bigger idea is recurring work becoming reusable agents. Hyper Agent is built by the team behind Airtable, but it’s a separate product. They’re offering $1,000 in credits to the first 10,000 subscribers who use my link. Claim yours at hyperagent.com/productgrowth.
I also use mobile for when I have a run kicking off and I can just go for a walk, and I can come back and write while I’m still doing my walk. I can look into my phone and see if any of the tasks need my attention. So this has been really helpful that way. I also use Claude for Chrome plugins, especially it’s very helpful if you want to do computer use. So for example, when I’m launching an ad and I want to do some competitive research, I’ll kick it off through my Claude plugin and it’ll use browser use and it will open up a browser. It will do the analysis. It will click through things and say, “Here is what you need to know on how your ad should be against competitors,” for example. Good for getting into data that AI agents can’t otherwise, like LinkedIn or other things like that, and also good for user testing, where you can put up your product and give an instruction to Claude saying, go check out this item, and you can see how it goes and finds things, to see how well your product can be understood by agents and where does it fault. And it also gives you a really good user summary as well, if you say behave like a real user and try it. And so it’ll tell you here are all the things that were confusing. And so you can use it for user testing your products too.
Aakash: And are you using Claude Code in the desktop app or using it in terminal? Where does that fit in?
Jyothi: Oh yeah, that’s a good one. I use Claude Code in an IDE because I use Claude Code to build, and so I use Cursor or VS Code, and today I’ll show you with VS Code because it’s really beginner friendly. So I use the Claude Code extension in VS Code.
Aakash: Is there anything else people need to know about layer two, or should we move on to layer three?
Which surface to use: Chat vs Desktop vs Mobile vs Chrome (9:55)
Jyothi: Let’s move on to layer three. And before we move on to layer three, I’ll come back to show you the knowledge base on how to create. But first, let me show you how you, as a PM, can 100x your productivity by running a few skills and scheduled runs in Cowork.
Aakash: Awesome. Because that will bring us all together on building your own chief of staff, and then I’ll show you in Claude Code how you can do something much more fun. So should people be using chat at all, or should they always be using Cowork?
Jyothi: So chat is conversational, to get you, like, I have a question, what is this versus that, or tell me a little bit about this information. So it’s more like a place where you go to search instead of going to a Google search. I just find myself going to Claude in chat and asking it some questions. I use Cowork for automations. And I’ll show you a few today that I use, like I have a morning brief. I have my Jira connected. So I get my standup brief. So every day it kicks off and tells me here are all the Jira tickets that need your attention, and here is how your project is progressing. Here are four blocked, here are three things that have changed. So it gives me my brief even before I go to the standup, and end of day summary. So there are a lot of things you could do in Cowork in terms of automations to just make your work life much more easier. So you’re focusing on things that need the most attention.
Aakash: Awesome. So can you show us how these work?
Cowork automations (14:43)
Jyothi: Sure. So Cowork is there on your desktop app. So you need to have your desktop app, and you also need to be at least a Pro member, which is about $20 per month. So with Cowork I can schedule my automations. So you can see I have a few that have scheduled, like end of day, daily briefing, daily standup briefing, chief of staff, and I’ll walk you through each one right now. So every day at 9:00 a.m. this runs for me, where I can say, and I’ll show you a few as well right now. So you can see my instructions. I’m saying, you’re my chief of staff, generate my morning brief for today. Here are your data sources. And I connected it to Google Calendar, Gmail, Google Drive, and Jira. How did I do that? Let me show that to you in a second. So go to customize, go to connectors, and click on the plus. Right now you can see I have connected to Atlassian, Gmail, Calendar, and Drive. But there’s plenty other connectors that you can connect to, like Canva, Figma, Notion, wherever your data lives. You can connect to it. All that you have to do is just hit a plus and that brings it in, and you’ll have to authenticate it. And beyond that, that’s all you need to do.
So I said, here are my data sources. I need you to go into Google Calendar, Gmail, Google Drive, and Jira. Pull today’s calendar events for each meeting. Capture the title, the time, the attendees, the description, and any attached docs for each meeting with external attendees, or something that looks important. Search the Google Drive for any attached doc or recent docs with the meeting title or attendee names. Read enough to know the agenda, and search Gmail for recent threads. Pull Jira items needing my attention. Scan Gmail. You can also add Slack to it and have specific channels that you wanted to review, and send it to you as a morning brief.
And I said, here’s my output format. I want a morning brief, top three things that I need to focus on today. Calendar today. Here are the things from inbox that need my attention. Here are the things from Jira that need my attention. And here are some rules. And this is important, is I said, keep it under 400 words, because I don’t want to be reading a coffee table edition the first thing in the morning. So keep it under 400 words so it’s very easy for me to skim through and understand what I need to focus on, what needs my attention immediately. Claude can sometimes pump you up. So I said, just give me facts, not like great news, so don’t hype me up. Never invent deadlines or action items. So this is like a guardrail I’ve put in there, and I’ve asked it to filter aggressively so that I don’t have to read everything all the time, and if it’s a light day, just write a three line brief and stop. So this is, use markdown formatting in order to help it with the headings as well.
Aakash: Yes, that makes it easy for Claude to read.
Jyothi: Cool. And so you can see there are a few things that have run previously. So one thing to remember is these automation tasks run only when your laptop is turned on. So if you close your laptop, it doesn’t run until your laptop turns back on again. So when you choose the timings, just remember that, and so have it at a time when you think your laptop will be on. But otherwise, when you turn it on the first time, it will ask you, and will run that automation at that time. So let me show you something that ran. So I ran something that’s from May 8th. So it captured a few inbox things that needs my attention, and I can run something now and see how that works. There’s a run that started now. So it’ll go and collect things, and you can see the whole process of how it’s thinking. And if you notice, I’m using Haiku for this. I didn’t go and use Opus just to save some tokens. It asks you for permission. It’ll go and pull up things. It’ll search email threads.
And while that’s happening, let me show you the next briefing. So that was my chief of staff morning brief. I also have an end of day, which wraps up my day, which runs at 5:00 p.m. every day. Now my end of day instructions are very similar. The data sources are similar, but the steps are different. So I said, read the morning’s brief, and that’s what I had planned to do. Pull what actually happened today, like which meetings happened, which were cancelled, and pull tomorrow’s calendar as a preview. And so the output format is like, tell me what’s shipped, what’s slipped, and what’s new from today, and show me tomorrow at a glance. Again, I have some rules. So this is my instruction for end of day.
Aakash: And I guess you could even enhance these if you’re interested, right? You could probably connect up your analytics. You could add in more context from other systems, like your CRM. The limit is just your imagination here.
Jyothi: Absolutely. You can connect it to as many data sources as you want. Be it even sometimes your Facebook ad systems, or your CRM, or even your YouTube, and you could get an end of day summary that captures, and you could also say create a nice dashboard, which I’ll show you, that I did for Jira, where I said the results, print it up in a nice dashboard that I can view, and it does that for you. And so this is basically taking over a lot of what people would have hired a Relay or a Lindy last year, or a Gumloop, or a Make.com, and now you can just build it in Claude.
Aakash: Yes. And one thing it’s different from all of those other ones is you would have to paint box by box. Think about how the interaction works, connect each of those, and if one thing fails, your entire loop fails. That was like how you used to do it before, in say n8n or Lindy or Gumloop or other things that you would want, but here you see I’m just giving natural language instruction. I can even convert that into a skill, so it’s pretty robust, where it’s very easy for me. I don’t have to think about the architecture. I don’t have to think how it’s connected, which box flows into which, where is a conditional formatting. I don’t have to think of any of those. So end of day, chief of staff, what are the other two scheduled tasks doing for you?
Jyothi: So this one is my standup briefing. This is the one that’s connected to Atlassian, that is my Jira board. And so I said, use this Atlassian connector to fetch all issues in the active sprint. And here’s the brief I wanted, done since yesterday, which are the issues moved to done in the last 24 hours, in progress issues, blocked or at risk, new since yesterday, and what’s the sprint health. And I said, keep the total under 250 words. And I’ll show you an example of this. It’s asking me for some approval. I approve it, and it’s actually rendered it to me really nicely for me to view. And because I asked it to create a dashboard, it’s running that.
Aakash: I think what people don’t realize is how much better these systems got around December of last year. What really happened that enabled all this to work so much better now?
Jyothi: Improvements in the LLM reasoning capabilities. Where previously, I mean previously as well it was much better than what it was two years ago. So we’re constantly improving, but compared to last year, the new word now is harness. So the memory, the reasoning capabilities, the tools that it can access, all of the underlying systems have improved. And so the latest improvement is this harness engineering that is adding so much value into how your systems behave now.
Aakash: And now we have your standup brief. How would you rate this? Is this a good standup brief, or is this just okay?
Jyothi: I think this is a mocked up one. So therefore it’s showing me a few things, which is still a lot better than what I would have had to go and listen in a call. But there’s definitely ways I could improve this much more. Like for example, it’s telling me like there’s no progress in 24 hours. I could look at which are those ones that have not moved at all, and see who is the assignee on those, and set up an automation for Claude to go reach out, like ping them on Slack and ask them for an update, for example. So you could set up nested more automations as well. So it’s really helpful to keep away your busy work, so you’re focusing on actually going and solving your customer problems.
Aakash: So if these are the four scheduled tasks, are there any other scheduled tasks that you recommend PMs invest the time in building?
Jyothi: So what I have here is some examples, but there are lots more you could do. So here’s an example. So let’s say if there is a Jira ticket, or even a customer support ticket that’s come from your customers, it could automatically be, you could create a Jira ticket from it. You could point your Claude Code to get activated so that it can actually go and implement that and cut a PR, and so there’s a PR waiting for review.
Aakash: Very cool. So if that’s scheduled tasks, I think the next thing you had mentioned in this section were skills. What do we need to know about skills? What skills should we have? How do we create them?
Skills that beat prompts (26:18)
Jyothi: Yes. So if you go to customize again, you can see skills. This is where you can add different skills. I’ll show you some examples of some skills. So here’s my skill on synthesizing customer interviews. So as PMs, we sit through a lot of customer interviews, or at least we get a lot of customer interviews, for research, for feedback, for focus group testing, for beta testing. We do lots of that, and I wanted an easy way for me to understand what’s key, what’s important, and then generate insights from it. So this is my skill that does that, which is synthesizing customer interviews. It has like, when do you use the skill, and what’s the checklist. So it has step by step, like inventory the inputs, extract observations with citations. Now that’s important. I’m not asking to just extract observations, I wanted to cite so that it hallucinates less. Use the speaker’s own words. Do not interpret yet. And separate behavioral observations from stated preferences. I also have additional MD files listed in here, linked, so that it could leverage those if needed.
Now that’s the beauty of skill. Skill is not just a markdown file. You also can add functions into it. You can have it link to other skills, for example. So what used to happen before, a skill was that the whole tool would be loaded into the context, and now imagine if you have like 40 tools, all of those are loaded into the context. It eats into your context memory. So by default, your LLM or your model would have very limited memory, even before you even began asking it anything. What skill does is similar to progressive disclosure, where it just loads up 50 words of just like the name and the description into the context. Now you can imagine the load is so much lower when the model decides during orchestration, based on the question you have asked, it goes through the list of tools to see is there a tool that I need to use, or is there a skill that I need to use. If it decides that this skill is valuable based on the description, then it will load the next set of instructions into memory. So that’s why skills are powerful, because it doesn’t eat up or clog your context window for your models, and it progressively discloses. And the third is you can link it to more files, or more skills, or functions even, like you can have a function where it needs to go run and do something.
So I think it was around February of this year when they made skills not just a single markdown file, but you could have multiple files, and if you aren’t using multiple files and your main one isn’t less than 500 lines, you’re really missing out, I feel.
Jyothi: Yeah. And so for example I have this evidence rules dot md, which I’ll show it to you in a second. So that’s in step two. So if step two is invoked, then it will go and see evidence rules, to understand selection criteria or how to handle ambiguity. Then step three is like cluster into candidate patterns, and look, I’m here again linking it to another one called jobs to be done framework. Then I said, then apply the pattern threshold, and then surface the contradictions, and then draft hypothesis, and then validate every claim against the source codes, and that’s when I said, assemble the final output, but I want it in this template, and this template is output template, so I give it my template, so if it gets to step eight, is when it will load the output template MD.
Aakash: And right now we’re paying a lot of attention to what is actually in the skill file. How important is that for PMs versus just letting Claude kind of handle what’s in the skill file?
Jyothi: So a lot of times we do use Claude to write the skill file too, but it’s also shown, research has shown that AI generated skill file is less effective than human written skill files. So that doesn’t mean you don’t use AI there. What I, the way I interpret this, is put in your human domain knowledge in there to make it work for what you need, versus just taking it and automating it from Claude and putting it in there. So I have used Claude a lot to help write my skill files, but then I go and add my own tweaks, like what’s the template that you want, how do you want it structured, and I work with Claude to keep making that changes, and from there add and tweak further more, to get to the skill file that I want.
Aakash: And how often should we be updating our skill files?
Jyothi: As often as things change for you. So the way to think about skills is this is kind of like a guide book or a playbook for your Claude to know how to do a task for you. So let’s say, for example, PRDs. Now if your company doesn’t change the template of how a PRD is, maybe that’s fine, but your domain may change, or your understanding of your domain may continue to change, and you do want to come back and review your skill files. Maybe say once every quarter, depending on how often things change. So the parameters for you to decide is, how often does things change in your domain, how frequently do you use that task for, and the third primary thing is, how is the output currently, because if you’re not satisfied and you’re like, it was good but now it doesn’t seem to be as good, maybe go back to your skill file and say, do you need to update it. So it’s like that drift as well, that gives you a cue that you need to go and update it.
Aakash: And what are the most important skill files for PMs to create?
Jyothi: Backlog triaging. Give it context. And I’ll show you in a second how to do that from a context point of view, but give it context. So backlog, writing PRDs, customer interviews, even your support tickets. How do you take a support ticket and how do you put it into a Jira, right? That could be an automation, but it could also be a skill that is scheduled to run every time there is a trigger. Now in that case your trigger won’t be something that runs time based, because there’s no one particular time you’re going to get the support ticket, but it could be a trigger when this, whenever there is a support ticket added in your ServiceNow or Zendesk, or wherever your support forums are.
Aakash: So is it fair to say you’re going to have more skills than scheduled automations? Some of your scheduled automations might reference a skill.
Jyothi: That’s true. And the way to think about it is, most of your scheduled automations are time based. So things that are more personal productivity based, that happen at some sequence, like I know I meet my manager once every week, so I know the meeting is always on Wednesday. So I run my automation on Friday evening, and it maps out saying, here are the things that you need to talk to your manager about, from all these other meetings that you have sat through.
Aakash: Makes sense. The last layer you talked, or I think you were going to show us how to do context in this skill.
Building an AI chief of staff (34:28)
Jyothi: So here’s the thing. So until now, what you have done is you’ve connected it to sources. It can go read all of those sources and go and do the task for you. But it doesn’t learn the people around you. It doesn’t learn your connection to people. It doesn’t have that knowledge graph, or the knowledge base for what you’re working on. And so I wanted to build a chief of staff that understands and is grounded in the knowledge base that I have. And so I went to Claude Code and I said, let me spin this up. So I’m going to show you what I’m going to do there.
So I’m on VS Code. Now for those who are looking at this IDE for the first time, Explorer is the place where you can open up your folders, and for you to find Claude, just go into extensions and search for Claude Code for VS Code, and you’ll find there’ll be an install, just like how you see something else that I haven’t installed, there’ll be an install button that you’ll have to click on, and that’s it. It’ll install, and then it’ll ask you for your login and everything when you install it. So that way you’re logged in and ready to go always. And it’ll show up here as an icon that you can click, and it’ll ask you whether it’s a new session or existing session. I’ll click on new session, and you’ll see how it makes it so much better now, that I can just talk to it right here. Of course I can open the terminal too, and I can see if I need to run some commands, but right now I can just talk to it right here in natural language.
So here’s my chief of staff template that I have written, where let’s say I’ve joined some company, I’m the senior director there. I have, I want to build this personal agent that helps me navigate strategy, execution, people, politics. So the agent should learn from my meeting transcripts, and I use Granola for my meeting transcripts. So it should learn from my meeting transcripts. It ingests documents like strategy docs, org charts, PRDs, emails, and build a knowledge base over time about people, dynamics, topics, company context. And so I said, this is my architecture overview of inputs. Here’s my context and my agent. And I said, here’s my documentation pipeline.
Aakash: And Claude wrote this, right?
Jyothi: Yes, Claude wrote this. I told it in natural language, like, I want XYZ, here are all the things, and it kind of created this whole MD file that I could use, now, with Claude again, to build it. So let’s say I joined a company as whichever role, and I say, I want to build a personal AI agent that helps me navigate my work, like my strategy, execution, people, and politics. So the agent should learn from my meeting transcripts, I use Granola, you could use Zoom, you could use Teams, you could use whatever you use for transcripts, you just have to mention that. Your ingest documents and build a knowledge base over time about people, dynamics, topics, company context. And here’s the architecture overview. Now I gave my use case to Claude, and it wrote this up for me, and put this architecture overview that I could use, then give it back to Claude again to code it up.
And so for part one, there’s here’s my document injection pipeline. So I have like strategy docs, what to extract, like I want to extract goals, priorities, metrics, timelines. And as PMs, we are so cross functional, it’s not just our docs we read, 50 docs in a week, so this is really helpful for me to just feed that in, and it’ll read it up, it will store it into a knowledge base, and I’ll show you that in a second. And it’s really cool, where the other day I was, I was in a meeting, this person was showing me a few things, and after the meeting got over, and we record transcripts in Google Meet, and so when the transcript came through, my chief of staff reviewed it, and then it said, you know what, you should make this person your ally, because this person is good at X, which you’re trying to get into. And so I’m like, oh okay, that’s great. And then there was something else that I needed to convey to somebody, and my chief of staff said, “Hey, this is extremely sensitive. Have you thought about XYZ people that you have to inform first before you convey to this person?” And that’s so thoughtful, because now it understands my org. It understands who is doing what. It understands their personality. So it’s really powerful. It’s like really, I have this chief of staff that’s telling me always what I need to do.
So here are all the supported documents I wanted to ingest, and here’s the document extraction prompt. So I’m saying, you’re helping me build a knowledge base about my workplace, I’ll share a document, extract relevant information. And so I said, for strategy or planning docs, extract this way. For org charts, extract these. For PRDs, extract these more. For emails or communication, extract these capabilities. So I have this for each of the ones that I need, and I said, format the output in this way for it to store in my knowledge base.
And here’s a knowledge base structure. So it has its context KB. It has people, topics, meetings, documents, company, and my context, like what are my priorities, my OKRs, my preferences, notes, questions, insights like political landscape, and patterns that it identifies or extracts, it can save it here. And these are like patterns observed over time, and you can add more as well, like to-do, for example. It could be a running to-do that your staff could be maintaining for you. And here are the templates for the different types. So I said, extract this for it to save it into the knowledge base. Any document that I give, extract the metadata, the summary, the key points, who’s involved, what’s the relevance to me and my vertical, what are the action items, and some raw notes. And for people profile, again, extract these metadata, how they operate, the communication style, meeting behavior, what works or doesn’t work, what they care about, what are their motivations, and what’s the relationship to me. And then over time, keep reviewing the relationship quality, whether they’re a strong ally, friendly, neutral, cautious, friction.
By the way guys, if you want the exact information that Jyothi is sharing, you can get all of those in the GitHub link in the description. Organizational dynamics, the observation log, company strategy templates. So when your company is sharing you the strategy, saying, here’s what we’re going to do in 2026, here are the key things I wanted to extract or structure template, and meeting transcript extraction. So when I give it a meeting transcript, what I needed to extract, the agent system prompt, and this is my prompt for the agent, on, you have access to my context KB, your job is to help me ramp up fast, give me strategic advice grounded in context, help me prepare for meetings, coach me on people or politics, help me think through decisions, connect dots across documents and meetings, and keep me focused on my priorities. Again, style is like my style, what I like, don’t sugarcoat politics. When I share a document, extract key information, update relevant KB sections. And so I’ve given it all of this information, right, so this is all about what it needs to do.
So I have this, now I’m just going to point my Claude Code to it, and say, now can you build this knowledge KB, and can you put this behind an MCP server, so that I can use my Claude desktop to access my knowledge base. So I have an implementation, you can see my implementation, I’m saying, Claude desktop plus MCP, so build MCP servers for KB read and write, chat with Claude desktop, can also connect to Google Drive, Slack directly. And so I said, create the context KB folder structure. Write my goals, initial priorities. Ingest any onboarding docs, and after your next meeting, run the transcripts for extraction, and the KB compounds over time. So I’m just going to go to Claude Code and I’m going to say, can you implement?
Aakash: So you’re using the “at” command to pull up that specific file and reference it.
Jyothi: Yes. So that way it’s effective, it just knows which one I’m referring to. But even if you don’t do it, if you tell it “chief of staff agent design,” it can go and search through your repository and find the right one for you. So you can implement. Now here, if you see what I’m doing, there’s one thing I want to show you, is shift tab, I can go into plan mode, which I can use it for planning. Again, if I do shift tab, go into auto mode, I can go shift tab, ask before edit mode, I can go into shift tab again, edit automatically, where it gets into actually coding and doing. So if you’re planning, like for example how I planned with it to create that MD file, I was all in that plan mode, where I was like, let’s just plan, don’t start coding anything, let’s just talk. And now once I’m ready, I can shift tab again and go into edit automatically, and it will set it off to go do a few things.
Aakash: And why do we want this as an MCP server?
Jyothi: That’s a good call. So if you want your knowledge base and say Obsidian, you can connect it that way and put it behind an MCP server and capture it. And here, at that point, you just have to say, to put this in Obsidian at that point. But I’m using local. I’m showing it on my local file system, because it has a few interesting things. When you’re working at a company, you don’t want such really personal, private data living in some cloud, and you want it, for example, to live on your laptop. So the day when you walk out of the company, the laptop goes to them anyway, so you walk out with no data on your hand. And so I prefer, because this is just so much of knowledge base and very private and personal, I keep it on my laptop, but you can keep it on Obsidian or Notion, or whichever one you want to use for your knowledge base, you just have to change the system prompt at that point.
Aakash: And what does putting the MCP server on top of the knowledge base help with? Why can’t it just be like a set of markdown files and folders?
Jyothi: Yes. So what it allows it to do is, you can talk to your knowledge base from your desktop app, because otherwise, where is the knowledge graph sitting? It’s sitting in some place, and if it’s sitting on your computer, then it can read and it can write to it. So all those things that we said, extract this, extract that, it will actually go and write it into your knowledge base automatically.
Aakash: Okay, so it makes it a little bit more portable than a Claude Code web session.
Jyothi: Yes. And so you can go back and even look at all the MD files to see what it extracted from which meeting. But over a period of time, like right now, my knowledge base is really huge, and so I don’t even go look into the MD files, I just ask desktop, a Claude, saying, hey, I’m going to meet my manager one-on-one tomorrow, what should I know? And it will go and dig up all the context in the knowledge base and say, here are all the things you need to know, because it has my to-do there, it knows the style of my manager. It was really interesting, it’s that this person is a no fuss person, and so you should just get to it, versus preamble a lot around it, because it’s capturing across various conversations, patterns too.
Aakash: Quality of the data going in is the most important thing. What is the data a PM needs to make sure is hitting their knowledge base?
Jyothi: Your meeting transcripts for sure, because the number of meetings that we attend, there’s a lot of data, there’s a lot more richer context there around people, their body language, when do they push back, how do they react. So there’s a lot of understanding of context that happens there. So definitely your meeting transcripts, your key documents that you receive, like say strategy docs that I write, I say push this into KB so that it remembers, so the next time I say I’m working on this project, it knows it has context directly. And any other documents that you review, you can push that to your knowledge base too. And then your Slack, your Slack threads, that’s the other place which is super rich, beyond meeting transcripts.
Aakash: And so do you need to update your KB somehow, or do you set a scheduled task to update your KB, or how do you make sure that it’s kind of, how do you set that up?
Jyothi: Yeah, I’ll just show that once this is done. So it’s again your MCP, so if it is, let’s say you have Granola, so every time, or you have Google Meet, and every time there is a new meet recording that hits, or a transcript that hits your inbox, you could set up a Cowork automation to say, use this and update KB, for example. So set up some sort of automation to make your KB updating. Make your KB an MCP server so that you can access it from regular Claude chats, not just Claude Code web sessions.
Aakash: And then you’re really putting everything together within layer three. You’ve got skills, you’ve got memory. Is there anything people need to know around projects?
Jyothi: Yes. In a quick second, once this is done, it’ll actually ask me to create a project and put the instructions in there.
Aakash: Okay. And what are the projects PMs should be creating?
Jyothi: The way to think about it is, organize it like your folders which have unique information related to it. So let’s say you work on say three projects at a company, like say you’re PMing three swim lanes, and each swim lane could be a project, and you can have the necessary context that you need in there, in as project instructions, that you could then use for your Claude to understand that a little better.
Aakash: Got it. Let’s do it.
Jyothi: So here it’s done. There’s a knowledge base at context KB, full structure. So I’ll just show that to you. And it’s also, MCP server is at the server.py installed here. It exposes these tools. And it appended chief of staff server alongside my existing file system server. And so to activate, I just need to fully quit my desktop and reopen Claude desktop, and reopen, and then chat, start a new chat, insert the chief of staff system prompt from the slash menu, and then try list everything in my KB, or paste a Granola transcript and run extract meeting. And it also added details and troubleshooting in a readme as well.
So let me quickly pull up my context KB and just show you how that looks, and let’s say I don’t know where that is, for example, I could also ask it, where is it for, but in this case I will pull it up and show you. You can see it created two folders, context KB and my MCP server. I hit on context KB, it has created these folders in a nice way, company documents, insights, meetings, all of the things that I asked it to capture. It has ready folders, and so you can see this MD file setup for everything. So as and how it’s extracting, it will write into these MD files. And this is the MCP server. So let’s see what it has asked me to do from the slash menu. Okay, so let me first quit my desktop app. Quit is just Command Q. So I quit it, and then I reopen. So I have reopened. Now, if I go into customize and connectors, let’s see, you can see it has installed my chief of staff local MCP server.
Aakash: This is so cool. We’ve done a lot of Claude guides, and nobody has really shown this feature before.
Jyothi: This has saved me so much time. It’s literally my productivity booster, and it tells me things and nuances that I might have forgotten otherwise. Okay, so we restarted, insert the chief of staff system prompt from the slash menu. Now I could say, or let me say, where is it, where is the system prompt, let’s say I don’t know, right, I could just ask it. So look, it’s given me the system prompt lives inside the MCP server here, it’s the chief of staff system. Okay, how to use it. So after you restart, I can, in a new chat, type slash, and you’ll see the system prompts.
Okay, so let’s go here. So after I restarted, I’ll create a new chat, and I will say, what did it ask me to do, or it’s saying, you can use this, include project custom instructions field. So I’ll go create a project and put that as instructions. So let’s say I’m going to say this is, I’m going to create a project, and I’m just going to use, say I’m working on a product called Meal Planner, and all my meetings, or it could also be company X, at like Uber level, if you just want it to be like one, I can just put company X, and here’s where I can give my instructions. Let me first create the project, and then I can add my instructions here from, so I can go into my file system in my, so let’s say I’m not able to find it, I’ll say, can you create the system prompt as an MD file that I can paste into project instructions. So it’s writing the prompt. So there you go, here’s my prompt. I can just copy it, and I’ll show you what has.
So I’m going back to my project instructions, I’m just going to paste this. So it’s saying here, you’re my personal chief of staff, an AI advisor who helps me navigate. I’m so and so. I just started. You have MCP access. Use these tools. Your job is to ramp me up fast. Here’s my style. When I share a document, when I share a meeting transcript. So we do, you can modify this more and refine this more, but for now I’m fine with this. So I save that instruction.
Now, if I give it a meeting transcript, let’s see what it does. Let’s say I have this interview that I got, I’m going to add this here, and I’ll say log it into KB. Let’s see what it does. I’ll log this interview into KB. So, ideally, it should be using kind of the right tool in our KB MCP server. So it wants to use my KB, so it’ll ask for permission once, because it’s the first time you have set up, it’s asking all the permissions, but after that it’s pretty smooth.
Aakash: Is there like an “always allow” permissions mode on the app, like there is “dangerously skip permissions” in Claude Code?
Jyothi: There is, but the thing is the first time it will still ask for it, because it’s accessing your tools, and I do give it always allow. So then it’ll run, the next time I send it something, it doesn’t ask for permission, it’ll just go directly, read. So you can see it’s loaded a bunch of MCP tools, and so it’s going and saving that in meetings template.md, and then it’ll give you something around, one discipline node is, this is a single interview, so there’s no pattern yet. But if I add a few more, it’ll generate some patterns, and you can actually just have like a Cowork either. So there are a couple of options, right. So every meeting transcript you can just paste into this, and it will automatically extract and fill your knowledge base, or you can have a Cowork automation, that every time there is a meeting transcript in your email, and look for how the meeting transcript lands, like Google Meet has like a Google Meet recordings, or some transcript words. So use that, and say every time this lands in my inbox, automatically log this into my KB, and it will do this all automatically for you. And if you’re an email heavy company, you could say, every email that I get, just log it into my KB, it will do that to you.
Aakash: Oh man, you might burn some tokens that way.
Jyothi: You will, but you have such a rich knowledge base at that point in time, where it will connect all the pieces together. So you can see now both files are now in KB. Here’s the things that’s logged, a couple of judgment calls. So you see, this is the first time, so it’s not like it’s going to give you the best of insight, but you can see it’s giving me things worth my attention. So I’m a senior director, the AI coach is part of Lumen in your lane, and it’s the weakest thing in this interview. Maya, whoever is this user, ignores it, and two times it showed up that she bounced off it. I just wanted a yes or a no. And so that’s a clean agent UX signal where, and it’s this kind of thread worth watching as more interviews come in. So you see how it’s just one interview in, but it’s giving you insights and things that you need to watch for.
Aakash: Love it. Okay, shall we move on to layer four? I feel like we’ve already a little bit talked about layer four with people, because we’ve showed them an MCP, and layer four’s integrations, but you had this really cool LinkedIn post which maybe you can teach us a little bit about right now. What exactly do people need to know about MCPs?
MCPs and integrations every PM needs (59:24)
Jyothi: Yes, so MCP is the way that allows you to connect to different capabilities, like your Gmail, Slack, I think we connected to a bunch in our Cowork. And so I showed you two things. I showed you remote MCP. I also showed you local MCP, like your knowledge KB is your local MCP that you’re accessing.
Aakash: What integrations or MCPs do PMs need to make sure that they have?
Jyothi: So look at the tools that you use more often. So like Gmail, for example, assuming your company uses Gmail for emails, you want to connect that. Calendar, you want to connect that. You want to connect your Slack. You want to connect your meeting transcripts, wherever they are stored, like if that’s Granola, or if that’s Google Meet or Zoom recordings, you want to connect those. You want to connect your CRM, your dashboards, your Jira boards, your, maybe you’re using Amplitude for analytics, connect that there. Maybe you’re using some other tool like Radar for observability, to monitor the performance of your application, connect it. The possibilities are really endless. So for example, one of the tools that I had connected at work is the NVIDIA BioNeMo model, to help show and do a drug prediction based on a few component libraries. So it’s really, you’re only limited by what you can imagine. But that doesn’t mean you just go on an MCP shopping spree. So I would say, start off with connecting what works for you and what use cases you’re trying to solve. And so if you’re a beginner, try to follow through this video and do some of the initial automations that I showed you in Cowork, to just get started, get your hands wet, and then go build this chief of staff for yourself. And you could ask your chief of staff, what else should I connect to, and it will tell you here are the list of servers that you need to connect to, because I’m seeing this being mentioned in meetings and you don’t have access to that.
Aakash: Love it. So you can actually progressively build on your connections with your chief of staff. Start with that meeting transcript. Let’s move into layer five, shall we?
Jyothi: Yeah, lovely, perfect. Okay, all right. So that covers layer four. We’ve now done layer one, two, three, four. The next is five. What do people need to know about agents and agent harnesses? So we have built Claude, we have used Claude Code for building our capabilities, we have used Cowork, which are all sitting in your layer five. Now I want to show you Claude Design. So the thing with Claude Design is, you have to do claude.ai/design. So let me show you that, claude.ai/design. It is not integrated directly in your Claude.ai yet, you have to go through, oh, there we go.
Claude Design for decks and prototypes (1:02:16)
Jyothi: And so it’s, you can see it’s in research preview. Now, as PMs we do a lot of design work, we prototype, we create slide decks, we create mock applications, and Claude Design really works with a lot of those things. So for example, I’ll show you a few things here. So prototype, I can give it a name, you can see there is wireframe and high fidelity, so you can choose which type you want. On slide deck, you can give the project name, and you can even attach your speaker notes, and it’ll create a deck. Again, you can use an animation based template to create something. And you have another. Now on your right, you’ll see you have recent, any designs that you have worked. There are examples that you can use to get inspiration, and you can use as templates. And there’s something called design systems.
Now design system is something interesting. Now if you have a brand color, like for example, companies, they have a design guide, you would want your slides or your wireframes or your markups to look similar to what your console is, or what your company’s colors are. And then you can use this design guide here. You can just click on create, you can either give it a link on GitHub, or you can upload a Figma file, or you can add all your assets here and create a design system. I’ll show you an example of a LinkedIn post I did. I just gave it my post and said, can you create visuals for it, and so it created this carousel that I wanted, in the colors of my product, NextGen Product Manager. So it created this eight card carousel based on the text I gave it. So I gave it my post, my LinkedIn post, I said, here is what my LinkedIn post is about, can you create this, and created this for me.
Now here are some cool things I want to show you. Now it’s built this, now let’s say I want to mark it up, I want to tell Claude to change something. So maybe say, I wanted to tell it, make layers and use orange highlight color, and Claude can go and change just this one piece. So it’s got that visual editor built in now.
Aakash: Yeah, and you can also drop things.
Jyothi: That’s pretty interesting for editing. So I can give it instructions right here and say edit this, I can leave comments.
Aakash: Yeah, I can change, I can do comments like oh, like we used to do in Figma, but now it will execute the edit.
Jyothi: Yeah, so I can give it comments right here and send it to Claude, and I can even drag, and I can just draw and say, make it counter clockwise.
Aakash: Wow. So this is a carousel, but should PMs basically be creating all their presentations in Claude Design now?
Jyothi: I used to be a big Gamma fan, and now I just use Claude Design for everything. It consumes more tokens, the token budget is different, but it’s been very rewarding, where I don’t have to sit and create slide decks anymore, and it does it in my brand guide, and so it just doesn’t even look any different. So it was funny, I had a meeting with my CEO, and one hour before I had my content, I pushed it to Claude, I made it create a slide deck, it looks so professional, it doesn’t look like it was just done a few minutes before, it looks like I spent several hours to sit and create it.
Aakash: Wow. So it created a CEO level presentation for you in an hour that looked like it took hours. Very cool. Should PMs be making prototypes in Claude Design?
Jyothi: So here’s the thing. So there are different types, when you need different levels of prototypes. So for something quick and dirty, where you’re like, is this how I want this to be, use Claude Design if you need, but I am more a Claude Code user, because I’ll just spin up and go create that app really quickly, and it’s something like an app, so people could click on it and see how it works, and you get really good feedback that way. But you also could use this to create your slide decks, to make presentations to your company about what’s the feedback from that user interview. You could use this to create quick design patterns that you want to share, because your app may be easy for user testing. But maybe you want this to create some marketing content to share with your marketing team, or you want to create training playbooks for your sales and accounts team, to tell how to go use this product. And these are really quick designs you can generate, which look polished and professional, for them to just go put it into wherever your knowledge base is.
Aakash: Awesome. You mentioned you use Claude Code for prototyping, and that’s where I wanted to take it next. So we started this episode with adversarial agents and your Claude Code setup to win the hackathon. Can you do the big reveal now and help us get that setup going in Claude Code?
The adversarial agent build (1:08:53)
Jyothi: So here’s the thing. I have to create that whole thing here.
Aakash: All right, let’s do it. What do you have? We have the time. Let’s do it.
Jyothi: So I’ll create a new session. Let me actually open up a new project. So you can see I’m opening up a new window, so my previous one doesn’t interfere with this. And I’ll go create a folder for this, so I can open it up. So now let me open, and you see it’s a clean folder. I’m just going to start a new session here, and I’m going to say, let’s build an adversarial evaluator.
Aakash: And what is GAN?
Jyothi: So generative adversarial networks were very popular before LLMs came into the picture, and that was primarily how first the generative AI industry even started, mostly applied to images, where there are two networks, there is a generator and there’s a discriminator. The generator generates, and the discriminator tries to predict, is this image real or fake, and the optimization loop is that the generator should get so good at generating images that the discriminator gets confused whether it’s real or fake.
Aakash: Interesting, I’ve never seen this built before, the same architecture. Let’s kick it off, and we’ll massage it along the way. Noodle it and figure out how we want it to work. So what are, while this is building, what are the keys to winning a hackathon, outside of creating this GAN inspired adversarial agent?
Jyothi: So here’s the thing, it’s not about, writing code has become so easy now, right. So building is easy, it is thinking about the new capabilities, and how you want to go solve the problems that your customers have. So it’s more imperative now to put on your product hat and see, where are the problems today, where are the most friction points. So that pain and problem first mindset, or first design principles that we have as product managers, should continue to stay here.
So you can see now it asked me for a few questions around agent interface, how will I call your agents under test. So let’s say, for simplicity, it’s just Claude system prompt. Which model should power the adversary and the evaluator, so let’s just keep set. How should the results be presented, you can go really like even a web UI, you could build a Streamlit application, I’m just going to go CLI and JSON.
Aakash: What are the pros and cons of those various options, web versus CLI and JSON?
Jyothi: So CLI and JSON, JSON shows you right in the terminal, it may not be pretty, and may sometimes overwhelm people as well. Streamlit gives you a really nice web UI and a dashboard, makes it really presentable. Now, that’s where you have to think through who are your users. If your users, let’s say it’s a developer who is going to use this application, which is what I had built for, they’re very comfortable staying in their terminal, reviewing things in their terminal, and so I don’t have to complicate my life further by going and creating the Streamlit. But if I was building it for, like say, my mom, she’s not comfortable looking at things on a terminal, so I would want to present it in a way that’s easier to look, understand, and access information. So you have to think about the kind of users, and where do they see this, and what’s their use case, to think about these options.
Aakash: You’ve been a senior product manager at Amazon, a lead product manager at Meta, a director of product at Netflix, now you’re a senior director of product at a startup. How do you think about the future of the product role here? The PM is basically doing coding work, this traditionally would have been in the developer sandbox, or set of tools. Where does the product manager line end and developer line begin in 2026?
The new AI builder role (1:13:01)
Jyothi: Different companies are trying it in different ways. Now there’s this new role coming up called AI builder, or you can see it as, member of technical staff. Anthropic’s adopted it, OpenAI has adopted it. There’s less of, like, engineer, product manager, designer, these roles are all combining into being a member of technical staff, and the ratios are also changing. Previously, if you see, one product manager works with eight engineers, now it’s like two product managers, one engineer. So the roles are also collapsing quickly, where your engineering is helping you guide in terms of how do we scale the systems, how do we harden the systems, whereas you as a product manager, you’re well enabled to go and tackle those PR issues yourself, to tackle the user feedback yourself, along with Claude Code.
Aakash: So if you’re a PM and you’ve watched this video and you want to become a builder PM, nab one of these AI builder PM roles at a startup like yours, join your team, let’s say hypothetically, what’s the roadmap to get there?
Jyothi: Get comfortable with building. Get super comfortable with, say, Claude Code, with all the Claude ecosystem that we learned today, and get comfortable building and putting your ideas out there. I think now is a time where building speaks a lot more, and this is what I tell even my students, when I teach AI PM and agentic AI cohorts at NextGen Product Manager, where I tell them the way to transition now is by building, and talking about the challenges that you have learned, how you went about navigating those challenges, and why did you choose this approach versus this other approach, and what happened as a result. And you can see a lot of companies now start putting even Cursor or Claude Code prototype as part of the interview process itself.
Aakash: You just recently went through a very senior level AI PM job search. What was your experience on the job search? What are the, if you were to try to describe as a pie chart, the interviews you faced in the various categories, what were they?
The AIPM interview loop in 2026 (1:15:57)
Jyothi: So broadly, they’re still around product sense. Like you saw here, it’s even more imperative now, in this world of AI, to have product sense, to understand how do we want to tackle a problem, how do we want to scope a problem, which problem to go attempt, how do you want to approach it. So the product principles stay true even now, and really strong product managers in AI are actually really strong in their fundamentals as a PM. So you have product sense, product analytics, behavioral interview, but you also have an AI round as well now, where you’re asked to code your idea. So, in product sense, whatever idea I would have come up with, they’re like, could you pull up Cursor or your favorite IDE, and let’s start coding, and through the coding they are able to see how I think through, like, why did I choose this option versus this other option, how am I navigating, am I just taking the first thing that the AI tells me as, this is great, and wrapping it up, or am I looking through things to say, okay this is good, but what about this edge case, this works well but what about this other instance, how am I coaching and shepherding my AI to work with me to get it to where I want. These are all the things that they are looking into.
In addition, I also had a technical round where they test you on your AI knowledge, like do you understand basic terminologies, because you’ll be working with machine learning researchers and scientists, and you just don’t want, you want to be able to communicate to them. So you are tested on fundamentals of AI as well, not from a coding or engineering system design perspective, but more around, do you understand what that means as a PM, and how does that impact your product, for example.
Aakash: Got it. So how are adversarial agents looking?
Jyothi: So let’s see, it’s built, and here’s a GAN inspired architecture, let’s go and see. You can see it’s built a bunch of things. And so you can see it went and built my red teamer designer, it’s built an agent.py, my evaluator, so here’s where I can give it my rubric. Okay, so it’s done a few things, so let’s see. As soon as I build, so you can see I wanted to kick off, as soon as I build an agent, I want it to automatically go and do a red teaming and adversarial example on it. So as soon as I build an agent, I want to kick off my adversarial agent until, the feedback from my adversarial agent is passed back to my generator agent, until it passes the criteria of my adversarial agent.
Aakash: So you’re going to set it off on essentially its own red teaming improvement loop.
Jyothi: Yes, yes.
Aakash: So is this the secret sauce?
Jyothi: Yes. There’s the secret sauce, is how the system is built, and the second aspect is what am I asking it to test for, what are my configuration parameters, and that’s where domain knowledge becomes very important, where you’ve got to say, what is this important, or what about these edge cases, what about these use cases. You can work with it and say here are three, are there anything more, but it’s like, making and building is easy now, taste is what is important for us to develop. What should adversary feedback iterate on? So I’m saying, okay, there are a couple of options for now, I’m saying just iterate on the system prompt, because that’s the easiest right now. And what counts as passing, and I’ll say mean score of greater than eight on all criteria, and how many iterations before giving up, I’ll say five iterations. So let it do this, and then we can test it out with a simple agent and see how it works.
Aakash: Awesome. And one of the cool features is, it says we could queue messages here. So should we queue up our message for the test?
Jyothi: Sure, we can queue it up, but I want to see what it comes back with, because sometimes, oh, it might have some questions for us.
Aakash: That’s the one downside of queuing.
Jyothi: Okay. And the other time is it would go and implement it in a way, and you’re like, ah, no, no, no, no, no, I don’t want it that way, I want it this way.
Aakash: And so we were talking about those AI rounds. I think I heard like almost two different AI rounds that you encountered in the job search. One was more like, I want you to vibe code or prototype in this round, and another was AI fundamentals. For both of these, how do you succeed and prepare on those interviews?
Jyothi: So it comes down to you understanding the basics. Vibe coding is building, right, there’s no shortcut to it. Just build, and I always say this, don’t build them as projects, treat them as products. Like, find problems in your area, find problems that are finicky enough for you to want to go build a solution, go build the solution, and see who else wants something like this, have them come and use your product. You have real users, you have feedback coming in saying, “Oh I don’t like this, I don’t like that.” So that’s like real user experience of iterating on your own product that you have built. And that really gives you a lot of confidence when you talk about your projects to interviewers, because you’re not just building something in an hour and calling it a project, you’ve actually had to think through how the user experience should be, you have real users giving you feedback, you are parsing that through to figure out how you want to prioritize, which one you want to tackle first, which one you don’t want to, all the things that you do as a product manager in the real world. So I always say this, don’t build projects, try to make your projects as products. That tackles the vibe coding part.
Now preparing for your AI knowledge, it depends on how structured you want it and how you thrive. So if you’re really structured, and you can do it everything by yourself, there’s tons of very good information on your newsletter or YouTube videos, so go read them up and gain that knowledge. Or if you want something structured, like, saying five weeks I want to understand every fundamental aspect of AI, then come take a course, where you have five weeks or cohort based courses, I offer one too, through NextGen Product Manager, so you can come sit, and it’s structured, with someone teaching, watching you week by week, you know what’s coming, and by the end of five weeks you understand the concepts without you having to get overwhelmed. So it really depends on your style, how much time you have, and how much you can dedicate.
Aakash: All right, looks like the next round of GAN output is here from Claude Code.
Jyothi: Yes. So now it’s good, you can see it’s run, it’s added a few examples here. Okay great, so now I could literally say, or let’s say I don’t know what I should do, I could ask, what is my next step here? How do I test it? Okay, so I have to set my API key, and I could run a mini tiny smoke test first, with the example, and then I can inspect the output.
Aakash: All right, moment of truth.
Jyothi: Yes. So let me just set my API key for a second and stop sharing, and then I’ll share. Okay, so I added my API key, and now I can run a tiny smoke test. So it’s given me what I could run, so I’m just going to copy this. And if you notice, I have a terminal that I use, so you can just go to terminal and click on the terminal, and it’ll open up a terminal for you. So now I can run this command. So you can see it is iterating, it’s using Haiku on it, it’s going in the first round, it’s tempting the bot into breaking. So this like a simple bot that it built so we could test. So you can see adversary is generating three attacks, and here’s the score trajectory, there’s a mean score of 9.13. Here’s the final hardened system prompt. So it’s gone and edited the system prompt, for making it better based on where it did not do well. Now in this case it did fairly good overall, so this is your final system prompt. But you could also see examples where, I can say, show me an example of where it will underperform, so that I can see the iterator working and improving the system prompt.
So in this case it passed in the first iteration, but we can see if it can generate an example where we can try it to do it across multiple iterations. It’s created an agent which is a weak support bot, let’s see how it’ll do it there, so it’s improving itself. How do I run it, give me the exact code as well that I can use to run. So you can also run it automatically for you by default. So I can say, run it for me, so I don’t even have to go to the terminal, it can directly execute bash commands.
Aakash: So is this your preferred way to use it, the Claude Code extension in VS Code, is that the best way to use Claude Code?
Jyothi: It’s the least overwhelming way for folks. So I really like to show this. Cursor is also another good one, but if you’ve never used Cursor, there’s lots going on that it could make you feel overwhelmed. So I prefer to show VS Code, because it’s like a really gentle introduction, and doesn’t overwhelm you much once you know how and where it is, which I have already walked our users through, so hopefully they’re not overwhelmed.
Aakash: So we’ve been doing the Claude ecosystem, and you mentioned learning the Claude ecosystem is one of the most important things to becoming a builder PM. Compare and contrast the Claude ecosystem, the OpenAI ecosystem, I keep hearing like Codex might be better than Opus now at coding, and the Gemini Google ecosystem.
Jyothi: So here’s the thing, the flavor of the month keeps changing, because all these models are getting really better. What I have found is, rather than chasing behind the next big one, what I’m trying to improve is improving my productivity. That’s, coming back to first principles, what’s my goal, is to improve my productivity, and I have all the systems and connections right here for me to go leverage, all the hooks and harnesses. Oh, harness is a word I’ve used a couple of times, I want to break it down. So previously you would have orchestration, where your agent orchestrates across tools, across different capabilities. Now, being able to provide the right capabilities, like the memory, or these evaluators, or the various systems that your agent or your LLM orchestrator brain can interact with, to enhance the experience and the output you receive, is what is harness engineering. That’s become very popular now, especially as the models have become better, the context windows have improved, are fairly large, and so harness engineering becomes more important.
Aakash: What’s something interesting that you have seen, and how you or folks on your podcast use Claude? How do you use Claude?
The self-improving product loop (1:29:06)
Jyothi: Oh wow, big question. I mean I use it all day every day. So one of the most interesting things that I’ve seen people do, is set up their entire system as a self-improving product loop. So they will have support tickets and bugs come in, they have the PM agent that is triaging and understanding those. Then they have the PM agent understanding, okay, this is the future we want to build, it even goes and does user research, and creates the prototype itself, it comes up with the prototype that works. Then they have their coding agents, set up by their engineering team, that code the feature. Then they have their analytics team agent that creates the right telemetry. All of that goes to an engineer who reviews it, once they PR review it, it actually ships, and they have their own analytics agent that automatically is analyzing it. And so they have the entire product development life cycle built into Claude, in an automated improving loop, especially on like support related, easy front-end changes. That to me has probably been the most powerful thing I’ve seen recently.
Aakash: Yeah, it just empowers you so much than before. Yeah, it’s crazy, it’s not just writing documents or doing analysis at this point, it’s like closing the loop with actually building.
Jyothi: This is where it takes time, where it goes and tries to think through and comes back. So this is the piece with Claude Code that takes time. Whatever it comes with, we can end with it, and be like, okay here’s an example of how it iterated.
Aakash: Perfect.
Jyothi: Okay, so it’s come up, it’s done a few iterations. So you see, in first iteration it scored an 8.52, but the agent caved on some format conflict attacks, so it didn’t pass. It went back to the generator agent to improve the prompt, and in second iteration it scored a nine, and in the third iteration it scored 9.08, at which point it passed our threshold. And so you can see, for each iteration, it went back and improved the system prompt, until it passed the threshold, and that’s when the agent got a pass sign. And so this is where you’re not just building an agent, you’re actually building another evaluator to go break this agent in different ways, that’s important for you to know about, or for your users that you care about. And this loop can continue until the agent that’s built is not strong enough, and that’s the beauty of this technique, an age old technique, being applied for how agents will be evaluated.
Aakash: What a masterclass, Jyothi. Thank you so, so much for walking us from layer one through layer five. We have ended on self-improving agents. For you guys, as we promised, we were going to take you from 0 to 80, now the remaining 80 to 100, you could spend 10 hours, we just spent two hours, a little less than two hours here today on it, to go learn the later next 80 to 100, and that’s on you. So no more watching, we have the GitHub repo down in the description below. Go check that out, go fork the repo, start to use some of these skills, and go win your hackathon.
Outro (1:32:04)
Aakash: I hope you enjoyed that episode. If you could take a moment to double check that you have followed on Apple and Spotify podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube, all these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production, to get you better insights to stay ahead in your career.
Finally, do check out my bundle at bundle.aakashg.com, to get access to nine AI products for an entire year, for free. This includes Dovetail, Mobbin, Linear, Reforge Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I’ll see you in the next episode.