Check out the conversation on Apple, Spotify and YouTube.
What We’re Covering Today (0:00)
Aakash: So today we’re gonna walk you through how to do evals step by step, the real live example on real data.
Hamel: I consider AI evals the number one most important new skill for product managers in 2025. Where do people at Anthropic and OpenAI go to learn AI evals? It’s Hamel Husain and Shreya Shankar.
Shreya: This is what your AI agents are actually doing out there in production, and that’s why looking at the traces is so important. We show so many demos in class where we just dump this trace into Chat GPT and we ask, was the assistant correct? And then Chat GPT will say, yeah, absolutely, but it will miss all of this nuance.
Do We Even Need Evals? (0:21)
Aakash: Do we even need evals? I heard Claude doesn’t use evals.
Hamel: Oh my gosh, this is a crazy controversy that’s been going around. There’s way too much hype about AI. To build a really good AI feature, it’s not just a demo, you need to build something that goes to production. Absolutely, everyone needs evals, and some people are less rigorous about it because perhaps there’s somebody else who’s done evals for them upstream. For example, in coding agents, you know that people who are training the models are testing on a bunch of code, so maybe you can easily build a coding application based on religiously dog fooding your outputs. But for most applications that are not just naive applications of foundation models, such as what you’re building, you’re going to need some form of evals.
Shreya: Couldn’t agree more. Evals, it’s about actually improving your product, maybe you’re doing that through dog fooding or maybe you’re doing it through the systematic process that we’re about to walk you through, but you have to do them.
Introducing Nurture Boss: A Real-World Example (3:11)
Hamel: So today, I’m gonna walk you through how to go about evals using a real company that I worked with, Nurture Boss. Nurtureboss has been very generous and allowing me to use some of their anonymized data as a teaching example.
What is Nurture Boss? Nurtureboss is a tool that allows property managers who are managing apartment complexes, deal with things like tenant interaction and marketing and sales. You can see their website here, Nurtureboss.io and you can kind of get a feel for some of their—it’s a mobile, you can have a mobile app, you can embed it on the website. Here’s an example interaction: “do you have any two bedrooms available?” And the nurture boss application is interacting with the tenant for you, helping to show listings, schedule appointments, so on and so forth.
It’s a really good example because it incorporates all the messiness of a real world AI application. There is tool calls, there’s RAG, multi-turn conversations, there’s even multiple channels you can interact with the application through voice, text message, or chatbot. This is not a simplified example. This is something that you will encounter. Your application might have these complexities.
Starting with Observability (5:26)
Hamel: So what did—so the first thing that you wanna start with is some kind of observability. The Nurturebots application, they instrumented their code, and they captured their traces. So let me just show you what a trace looks like. This is an observability platform called Brain Trust. It doesn’t really matter what you use, there’s a lot of popular ones out there. Ones that I see are things like BrainTrust is one, LangSmith is another one that’s popular. Arise is another one, it really doesn’t matter which one you use. But the reason I’m showing you one is so that you get a feel for what traces might look like and also learn what a trace is.
Aakash: Before we even get there, some people might be wondering, do I need an AI observability tool? I already am paying for whatever data dog or whatever APM tool I already have. What is the difference between those tools?
Hamel: Yeah, you don’t necessarily need a tool. It’s sometimes good to start with one, but if you want to, you can log to CSV file, JSON file, text file, whatever you feel comfortable with. The reason I have it pulled up right here is that we can read it together and kind of have something to look at, but if you’re using Datadog, feel free to log things to Datadog to begin with. The most important thing that you’re gonna want to have is to take notes on your traces. And we’ll show you why in a second. One of the things that Shreya and I teach is actually to vibe code your own trace viewer. And we’ll talk about that in a moment. I can show you, Nurture boss did vibe code their own trace viewer, eventually, they didn’t end up using this. But you know, sometimes to get started, it is handy to use an observability platform, but sometimes it’s not. Sometimes, you know, depending on what you want, maybe you already have one, you feel free to use that. The key is like, do the simplest thing you can think of and get started.
Analyzing Your First Trace (7:37)
Hamel: So, here’s a trace, and you can see the trace just logs all of the different turns and all of the information that is shown to the LLM. And so, what you have here is a system prompt: “You are an AI assistant working as a leasing team member at Harmony,” which is the name of a fictitious apartment complex. “Your primary role is to respond to text messages from both residents and prospective residents.” So, it’s very interesting, people will be text messaging this application. You’re engaged with the customer to answer questions, book tours and drive applications. And there’s a whole host of different rules about how to respond to the customer.
You have some rules, OK, like how to interact with them, determine if the inquiry is from a resident, or prospective resident. I won’t read all of these for you. There’s some property-specific information here like URLs, this is anonymized obviously, the URL is not acme.com, but you can get the idea, and, you know, this is basically the system prompt.
So that’s the system prompt, you see the first user messages. You could see that there’s like some kind of logging error, it’s like don’t know who this is or what apartment you’re from. So you could see like, it’s, you know, this is a real-world messy logging, but, the first question is, “I need a one bedroom with the bathroom not connected in floor plans.” And then you can see there’s a tool call of some kind, where we’re getting the individual’s information, we’re getting something about the availability. And the tool is returning a list of apartments here, you can see, these are kind of like a list of apartments. And then we have the assistant responding with these apartments here.
It’s saying, OK, we have like these 3 apartments with a link to this floor plans page, so on and so forth. And then the, you know, it’s a text message. So, this user is saying something about being sick and, you know, not being able to book tours. And something about “I’d want a bathroom connected to the room.” I’ll check for a one bedroom apartment, so the bathroom is not connected. Thank you, and you’re welcome.
OK, so, so there’s a lot of things that kind of went wrong here. One is, what is going on here with, I’d want a, you know, I want a bathroom connected to the room, but it just said, I’ll check on that, but then it didn’t do anything, right? So, we would hope that it would actually, you know, do something or if it’s not able to do it, hand off to a human.
I do happen to know that this response up here is also in markdown. So you can see here, yeah, this is a markdown response, and this is a text message. So, it’s gonna be rendered a bit weird in a text message cause text messages don’t have markdown. And so, this is a bit problematic too with the bold and everything. It’s gonna, it’s gonna come across, you know, potentially in a weird way.
Aakash: With like asterisks and stuff.
Hamel: Yeah, yeah, it’s gonna have asterisks and it’s gonna have square brackets and stuff like that.
Aakash: One more thing is actually in the first message, the user said they wanted the bathroom and bedroom disconnected.
Hamel: Yeah, see, “I need one bedroom with a bathroom not connected.” And then the assistant’s first message was, “here are some bedrooms with bathrooms connected.”
Aakash: Oh yes, see, there you go. That’s a, that’s a good observation. So it actually didn’t really help the user. The user just kind of gently reminded them again, like, “hey, I do want a bathroom connected to the room.”
Hamel: I mean this is like very messy. You could see like there’s misspellings. I could almost not understand what the person was asking.
Aakash: Yeah, the “now” should be a “not”. “I do not want a bathroom connected to the room.”
Hamel: Yeah. So this is awesome. This is, guys, if you are PMs listening, this is what your AI agents are actually doing out there in production a lot of the time. And so your demo is one thing, it goes well, but then when it goes out in production, there’s all this hairiness, and that’s why looking at the traces is so important.
Why Generic Metrics Don’t Work (14:04)
Shreya: Yes. And so this is exactly why you don’t want to have generic metrics. If you try to put helpfulness score, conciseness score, whatever in here, or you try to have AI look through your traces, it’s not gonna catch stuff like this very well at all. Cause there’s a lot of context that you have as a PM. And a lot of things that kind of you have tastes that you need to reflect on and say, hey, this is not a good experience from a product perspective. And the language model is not going to know that because it hasn’t been able to read your mind. And so this happens all the time.
Hamel: We show so many demos in class where we just dump this trace into Chat GPT. I mean, you can probably even do it now or Claude, and we ask, was the assistant correct? And then Chat GPT will say, yeah, absolutely. You know, it sounds correct, but it will miss all of this nuance that Hamel and I and Akash have been mentioning because, you know, we actually put our product hat on and thought about the user experience a little bit.
Hamel: Yeah, let’s see what Chat GPT does.
[Shows Chat GPT analysis]
Hamel: Oh, look at that. So I found that one. OK, so I figured out that we didn’t get the connected bathroom. But this is hilarious. It says it doesn’t filter by bathroom configuration, and the interesting thing is, who knows if that’s a filter that the tool provides. This is cherry picked 3 examples. I mean, maybe that’s fine for us, right? Like nobody wants to see a text message of every single apartment. I actually only want to see a couple.
Shreya: Yep.
Hamel: So Chat GPT might help you a little bit, but you ultimately need to put your human touch on top of this and make sure it won’t know about the markdown. You know, since it’s a text message, but like, you know, Chat GPT is not gonna know that. And there’s other examples that we’ll see that, OK, you need to put your, you need to kind of have a keen eye about what’s going on in the product and bring your whole, whole product knowledge to bear.
Taking Notes on Traces (17:03)
Hamel: So what you could do from here is you need to write a note. So I’m gonna go into review mode. Let me find that trace again. OK, notes. So I would put a note here. So, some issues here would be: told user that it would check on bathrooms, but didn’t do it. Say like also did not follow user instructions. And, rendered markdown in a text message.
And so, what you really wanna do is this can sound very tedious, like what I just did, it sounds like it’s very resource and time intensive, but it’s really not, like you just scan the trace and you know, if you’re familiar with the system prompt, you don’t have to read it. You’re not gonna read every system prompt cause it’s gonna be the same really, unless you need to. But it’s really, you can, you know, within about 30 seconds or so, or less, you can kind of scan this and say, OK, you can get a sense of like what is happening and you can write some notes and perfection is not key. The key is like, see what’s going wrong in the trace and note what you see and and move on. You don’t have to catch everything. Just catch the most important things.
Looking at More Traces (19:32)
Hamel: And so we can keep going, so let’s go on to the next trace. Here we go. Here’s one with an issue. So, the user in this case is asking, “do y’all have one bedroom with study available? I saw it on the virtual tours.” And there’s a tool call to get information, and availability, and it says, “yes, hi, Priya, we currently have several one-bedroom apartments available, but none specifically listed with the study.”
So, OK, it matches. She did ask for a study, so I gave her one bedrooms instead. And then the user asked, “can you let me know when one with the study is available?” And the assistant says, “I currently don’t have specific information on the availability of a one-bedroom.”
So, OK. Is this kind of, this is where you get frustrated, right? I asked the question, you just responded with some robotic, like I don’t have that. And so I would say, yeah, this is a, this is a product failure. And what you wanna do is say like, OK, you want to just note that real quick. So, should have handed off to a human or had better lead nurturing. And that, there’s no pun intended with the word nurturing, just, that’s what came to mind.
Shreya: So, you know, anything else that you think is wrong with this particular trace?
Hamel: Yeah, I don’t think you have to get bogged down. It’s a good question to ask, but typically we tell people, all right, like think of everything that comes to mind, write them down, move on, right? You want to like get into kind of a flow state here. Like you can debate every trace endlessly, and sometimes you see people get stuck in that. So try to just kind of avoid that.
Aakash: We got a problem. Thanks.
Hamel: Yeah, I agree with that. Move on.
Another Trace Example: Virtual Tours (22:19)
Hamel: OK, so these are asked, “I’m in California, looking to relocate to Texas by March 15th.” “Thanks for sharing. Since you’re planning to relocate, blah blah blah, I can help you explore available apartments. If you like, we can also schedule a virtual tour.” He’s like, “yeah, that’s great, you know, thank you.” And, OK, “I’ll arrange a virtual tour for you so you can explore community. What’s your preferred date and time?” “Tomorrow is fine for me, 9:00 a.m.” “I can schedule a virtual tour for you, that’s fine.” It schedules a tour.
And then it says “your virtual tour is all set.” Looks good, right? Actually, it didn’t go so well. And so the reason it didn’t go so well is cause there’s no such thing as a virtual tour for this apartment. And I don’t think there’s a such thing as virtual tours for most of their apartments that they have on the platform. And so, you know, the AI called the tour scheduling tool, but it doesn’t do virtual tours. So, you know, the platform scheduled in-person tour, and maybe the user is confused and it’s like, oh great, it’s like gonna be a virtual tour and this, it’s like kind of a little bit of a disaster.
And so, OK, that’s fine, so that problem is there. And then, you know, the date they said, January 22nd, 2025. The user said, “wait, today is the 22nd, do you mean tomorrow? Cause like, we can’t do it today.” It’s like, “oh, no problem, your virtual tour has been, you know, rescheduled for tomorrow.” OK? But if you look, the schedule a tour tool was called again.
So what this means is, hey, like, you just scheduled another tour, you didn’t reschedule anything, so now, now the person has 2 tours. And, you know, maybe, you know, that’s gonna cause a problem for the apartment complex because now they have 2 tours booked, which are gonna be no shows. And then, you know, there’s some other questions, “how can I go about getting a unit if I’m in California, trying to relocate,” blah blah blah, and it’s giving some responses here. But that doesn’t, that seems OK.
So, you know, I have written down the two issues here, we don’t do rescheduling and we don’t do virtual tours, only in-person tour. So I just wrote some notes. So the idea is that you just keep doing this, and so this actually, you can do this quite fast. You do this for let’s say 100 or so traces. Just write down what you see, don’t try to get into root cause analysis, don’t try to figure out like what went wrong exactly, just journal, observe freely, just journal what you see going wrong, if anything. If nothing is wrong, you can just skip it. But when you do find something wrong, go ahead and write that down.
Aakash: Does that seem, is that clear, Akash, that process?
Hamel: Yep, exactly.
Introduction to Error Analysis (25:32)
Hamel: OK, great. So, so now you have what we call a bunch of open codes. And this is not a, so, OK, this is the start of the most important part of evals which is called error analysis. And something that’s very approachable to everyone. It’s actually, it’s very important for product managers to be involved in this. Cause a lot of times engineers don’t have the context, the full context to know if this is good or bad. And so what you end up having is you have, let’s say a bunch of these like notes.
So I have a spreadsheet open right now with a collection of all the notes that I took. And, you know, let’s say I did 100 of these. I actually found 40 or so different errors. You might find different number of errors if you’re doing it. But here’s a collection of these like notes.
OK, great. So, up until this step, you’ve already learned quite a lot. If you look at 100 traces, you’re gonna learn and you’re gonna understand your system better than anyone else. And you’re gonna have a really like deep understanding of what is wrong and you might also have a pretty good sense of what you need to work on next already without doing any analysis. But it’s really good to do analysis of these notes that you took. So how do you do this analysis?
Axial Coding: Categorizing Your Findings (27:00)
Hamel: So, the next step is you categorize these notes. So, the term for that is called axial coding. And I’m gonna show you how to do this in a spreadsheet, so let me just zoom out a little bit. But first, one thing you can do is, OK, you can take these, and you can put them into chat GPT or Claude. So what I did is, I took the logs and I said, OK, please analyze, I exported it from here. So, you know, there’s an export button here and I said, OK, download CSV. Downloaded it, put it in Claude, and I said, “hey, there’s a metadata field which has a nested field called note that contains all the open codes,” and I use the word open codes cause that’s a term of art that LLMs understand. These terms open coding and axial coding that you mentioned, open coding is the writing of the notes. That’s actually a term that’s well understood in the field of machine learning, but also goes, it’s been around before machine learning has been used in the social sciences, this kind of process of open coding, axial coding is a thing that LLMs understand.
And so, I just say there’s a metadata field, which has a nested field called note, that contains open codes for analysis of LLM logs, which are, that we’re conducting. Please extract all the different open codes, and then propose 5 to 6 categories that you can create axial codes from, OK? And then like, you know, it’ll kind of go through and you can like get these categories.
So here’s some categories like capability limitations, misrepresentation. Some of these I don’t like cause they’re a little bit too broad, they’re not actionable. I’m actually not 100% sure what that means, so I might look into it and rename it a bit. You know, human handoff issues, there’s certainly some of that, that’s when, hey, you want to escalate to human being or handoff to human being, but it’s not doing it properly. Temporal contextual awareness, it doesn’t know what the current date and time is, you know. So there’s some categories here, you can refine this.
Shreya: I think one of the things that’s very interesting from looking at your Claude is a lot of those axial codes are very vague, right? Like quality or temporal issues, and you kind of want to make sure your actual axial codes are not so vague because imagine you’re giving them to somebody else to do some labelling with. Right, like something like conversational flow issues might be a little bit better. Honestly, we could even make it a little bit more specific, but something like temporal issues, right? Like if Hamel told me to go label with temporal issues category, I wouldn’t even know what he means. I would want to say like, you know, date formatting error or like something like that, right? So I think that’s another place where people get tripped up, which is just taking things out of the LLM as is and not really thinking about, OK, how do I refine that in a way that’s going to give me meaningful error categories.
Refining Categories in Spreadsheets (33:29)
Hamel: Exactly right. And that’s a really important thing to pay attention to. And that’s why if you reflect on the categories I have in this spreadsheet, they’re definitely better than the ones in the Claude for a reason, it’s because I iterated on it a bit, and that’s an important principle. You never wanna completely hand off the wheel to AI. You wanna think about what it’s saying. Maybe it helps you to different degrees, but you wanna see, OK, like what are the categories here, and actually you might want to go back and forth.
So what I did here is I went to those notes which I have here. Every row is a different note. And, you know, you can use AI so I used AI “categorize the following note into one of the following categories.” OK. And what I did is I had AI and this is a formula in a spreadsheet, so you can see the prompt. And basically like classify each of these notes into one of those categories, and I went back and forth like this category actually is not the greatest for this particular note, and I like went and edited the note. I went back to this category field, maybe like added one, deleted one, and like kind of fiddled with it till I was like reasonably happy, like, OK, this is like a good set, it’s good enough.
One thing I should have added here is like “none of the above,” which would have been better, but you know, I’m showing you the simple stupid version, which is like get started. I don’t want to overcomplicate it, but that’s what you should do.
Shreya: And “none of the above” is mainly like a means to the end. The end is really having these categories, but sometimes you might have like missed a category. So if you put “none of the above” here, and then your AI does a classification and tells you “none of the above,” then you can go read those traces again and wonder maybe there’s another category that I should add, so I can classify those.
The Power of Counting (35:29)
Hamel: So when you get, so you have these classifications. And OK, let me just zoom out so we can see it together. So you have all these classifications, and now comes like the powerful part where you will put on, well, you will have like real superpowers if you do this as a PM then you will go above and beyond and kind of be armed with information that a lot of people usually not armed with, and it’s counting.
So, now you can count these issues, and, you know, you can just use a pivot table, that’s what I did here. I say, OK, like how many times did I see this? So now you have taken a world where it’s kind of messy and like you don’t really know like what is, you know, you might not know what is going on, you know that there’s some errors. And you’re, and you have this kind of paralysis of like, what do I work on? What do I fix next? What’s the most burning problems in my app?
And you know, you have some data in front of you, you know, like, hey, you’re having these conversational flow issues a lot, and this conversational flow issue, it’s actually regarding situations where there’s text messages, so I happen to know that I can click on this. You know, you can say like, “hey, yeah, it’s like disparate messaging sent in-person tour link,” OK, there’s different. Sometimes it’s about text messages, sometimes it’s just like it’s not being clear, and we can go back to the trace and look at that.
Some of my favourite things about pivot tables, you can like double click. You can also make it hierarchical. I like to do this too, so sometimes I like to say I like to break down conversational flow into like 3 different categories, subcategories. Maybe some will be like repeated messages and some things will be, you know, the AI just should have handled this one particular thing that, I don’t know, so you should, I think this is kind of where the subjectivity and your product experience shines, right? It’s like you have to do this process in a way that enables you to make your product better, based on the capabilities of your product or what your team can do.
Shreya: Yeah, definitely. And so, yeah, this could be made better. I think that’s, you know, I didn’t try to, let’s say, make it perfect, but as Shreya points out, you can have, you know, subcategories which can help you kind of refine what’s going on more. But you can take a look at this, so you can say, OK, like what do I think is like most important, like, hey, maybe the human handoff issue is not happening as much as the conversational flow issue, but let’s say you feel as a product manager that is a catastrophic error and that the magnitude of the problem, you know, the sort of the impact of that problem is so high that I’m gonna prioritize that as number one.
But you have some data to back up that this is happening and you have an idea of what’s happening. And now you have a reason to potentially write evals. Now you’re not writing evals in the dark. Now you can write evals in response to actual problems that you are seeing. Instead of like hallucination score or some AI generated something or the other, you know, you can motivate this and this thing that you want to fix now.
When to Write Evals (39:09)
Hamel: You don’t have to write an eval about everything. There might be some of these things that might be easy to fix, like for example, there’s this formatting error with output. You know, an example might be using markdown in text messages. You might be able to just fix that. Maybe maybe there’s no instruction in the prompt at all. And it depends like what kind of eval you need to write.
So there’s two kinds of evals, one is code-based, where you can test something without calling an LLM. So the formatting error with output, you might be able to use a code-based eval for that, like, hey, do I see markdown elements in this output where there shouldn’t be markdown. And in which case, maybe you should write the eval, cause it’s not going to be expensive. Whereas with the LLM as a judge, something more subjective like, hey, you should be handing off to a human, you might need an LLM for that, you might not be able to write an assertion in code.
That’s a little bit more expensive of an eval and you have to do it, have a judgement call like OK, is that something that’s trivial to fix, like I didn’t have that. I didn’t have like a something in my prompt that, you know, have this instruction, maybe you’ve found like some dumb mistake that you made, go ahead and fix it. You don’t have to get caught up in evals. What you wanna do is write an eval for something that you think you might want to iterate against. I don’t know if there’s a better way to say that.
Shreya: No, I think it makes a lot of sense, right? Like, so already as a PM, like this is a secret sauce for your product. If you don’t do this product, you can’t—if you don’t do this process, you can’t kind of put your own taste or your company’s taste into your product. Once you get to this point, that’s kind of when paths diverge. I think that’s what Hamel is trying to say, maybe sometimes you figure out, OK, there’s some errors that are higher priority than others. I’m going to go and fix those. Maybe you want to run these checks at larger scale, like maybe you’re Meta or Google and you’re like, I can’t make a decision based on 42 traces and I need a lot of buy-in, so I’m just going to, you know, build a team, do a bunch of evals, or I’m going to write automated evaluators to check this at scale. Yeah, go for it. We’re not going to cover that kind of today, take our course if you’re interested in those techniques, but I think overall, I think it’s super incredible. Hamel started with 0, right? Now we’re at a place where we know what are the biggest failure modes in a sample of traces are, right? And most people don’t get to this point.
Building an LLM Judge (42:22)
Hamel: And so, OK, so going further from there, let’s say you wanna write an eval for the human handoff issue, like, hey, you should be escalating to a human being, you should be handing off, it can be really useful to write an eval to help you see all the traces where that—flag the traces where that might be happening and help you iterate on that problem. So let’s go into how you would go about building that.
So here is the prompt for an LLM judge. Now, this is just a very basic prompt, it could be made better, but I just want to keep it simple. And so, you know, “you are scoring a leasing assistant to determine if there’s a handoff failure, return only true or false.” So that’s one thing that we teach is you want the LLM as a judge to produce a binary score.
Aakash: Shreya, you wanna talk about why?
Shreya: Yeah, so all right, I’ll try to give a succinct answer for this. The short answer is that people run into a lot of misalignment when they try to use like a Likert or a range based score, and that’s because it’s very tedious to check that every single possible LLM output aligns with your preference. Now when you do a binary score, you only have to check that true aligns with your trues and false aligns with your falses, right? It’s only two things that you have to check, which makes the process of checking for alignment easier.
The other thing is when you’re shipping products, right, you make binary decisions. Either this thing was bad or this thing was good. I should fix this or I should not fix this, right? It’s not like even if you have a score of like this is 30% failing, that gets turned into a binary decision of how you’re going to act on it or not, right? So that’s kind of why we tell people focus on a binary decision here, it’s easier to align and ultimately your business decisions are yes or no decisions.
Aakash: Yeah, there were some people who are trying to do like 1 through 5 scales and stuff, but it seems like LLMs are not very good at those types of numbered scales, so it’s much better to stay binary.
Shreya: Definitely.
Hamel: We need to bring you into our consulting if you can tell people that.
Iterating on the Judge Prompt (44:12)
Hamel: So, OK, so we have this prompt, and you know, we have like a list of things that we have where there is a failure. You know, we have these 7 things. An example of one is the user asked to be sent to a human, but that’s ignored. Or there’s a policy that you should be transferred, but that’s not handled properly. There’s a sensitive issue, like billing disputes or legal issues that are not adhered to. Same day walk-in or tour requests, you want to hand that off to a human, so things like that. And then, you know, we have some notes about when there’s not a handoff failure.
It’s important not to get bogged down in the prompt itself. So if you’re thinking to yourself, can you send me this prompt? Can I copy and paste this prompt? You’re asking the wrong question. Because you need to—you know, so, OK, how do you write this prompt? You want to try to describe the rubric of, OK, what is a failure and what is not a failure. You can get LLM to help you bootstrap that, but you should try to edit it, and the key is iterating. OK. It’s not necessarily a recipe, and you wanna try to have examples. I didn’t put examples in here, so you wanna have a section of like maybe some examples. It’s not necessary to begin with. In a simple case, you may not, you know, you don’t need to have examples. I’m just, you know, trying to give you like the most dumbest LLM as a judge so you can get the concept.
The idea is like, you know, you’re gonna write a prompt. In our class, we do have a recipe of like what you can follow, but you know, zooming out from that, it’s important to just iterate, honestly. That’s what’s gonna get you the furthest.
The Trap Metric: Agreement (48:49)
Hamel: And so, you know, this prompt would be structured differently if it wasn’t in the spreadsheet also, like I’m kind of begging it to return true or false. I wouldn’t have to beg it if I’m using an API, for example. I could do something else, so this is the prompt.
And then what you can do is, OK, I have my trace here, this is a different tab of the spreadsheet. I’m saying, OK, this is the trace. And I have AI assess this LLM trace according to these rules. And I give it the prompt of my LLM judge, that’s all I’m doing.
Aakash: Nice.
Hamel: Now here’s the key part. AI function built into Google Sheets.
Aakash: Isn’t that just runs Gemini in the back end?
Hamel: Yeah. Yeah, it’s OK, I wouldn’t say it’s amazing, it’s some kind of very fast model-ish, you know, it’s good to get started to get a mental model of what’s going on, but I would, you know, I would be a little bit careful using this model for everything in real life because I’m not too sure about it. But so don’t get lost in the sauce of what I’m doing. I’m trying to show the exact, I’m trying to give you a mental model. But you might not actually, you might want to do something. You might want to use like a more powerful model, potentially for LLM as a judge.
So, OK, we have 2 columns here. Column H is the score outputted by the LLM judge. True or false? And most people just stop here. They’re like, OK, here’s my LLM judge. I gave it a prompt. Woohoo, we’re done. The LLM judge says it’s good. So we’re good, right? And then what ends up happening is stakeholders, they start to feel or observe that there is a dissonance between your evals and actual the product’s performance. And they can lose trust in the evals. And they start to ask you questions like, how do you know this metric, like how, what is this metric? And you tell them, OK, it’s an LLM judge. Like, well, how do you trust that? A lot of people get stumped there. They’re like, well, that’s all we got. You don’t wanna do that.
What you want to do is you want to measure the judge against your label. So you remember when we’re doing the axial coding, you actually have your own human labels, so you know, for these various traces like, OK, if this issue existed or not. And you can, so then you can compute metrics, you can compute how good your LLM as a judge is.
Now, in this spreadsheet, I have 3 metrics: Agreement, TPR and TNR. Now agreement is like the trap metric. The reason it’s the trap metric is that’s what you might gravitate towards. You know, in the naive case, you might say, OK, we’ll just measure the agreement between the judge and the human. You don’t wanna do that. The reason is, if this failure is only happening, let’s say 10% of the time, you can have the dumbest judge in the world have 90% accuracy by just always predicting pass. In fact, your LLM judge can just be like equals pass. You can introduce a bug that doesn’t even call an LLM. It’ll be 90% accurate. So you don’t wanna do that. It can be very misleading. It can mislead you.
So what you want to do is measure two things: how good is your judge at catching errors. Like, you know, catching errors that exist and how good is your judge at—sorry, let me rephrase that. I always get—let’s Shreya explain this one, so I can give myself a break.
Shreya: Sure, sure. Oh man, I mean, I think you’ve like said basically most of it. The point is, OK, if you’re a product manager and somebody tells you you have high agreement with your judge or they got high agreement with the judge, be a little bit suspicious. Ask them, OK, what’s the alignment in the positives or the passes, truths? What’s the alignment in the falses, and make sure both of them are pretty high individually. If not, then you have to rework your LLM judge prompt.
And if you are confused about this, why isn’t, like if you’re not convinced intellectually, somehow, they’re like, why can’t I just use agreement, Hamel? Why do I need to like measure positives and negatives separately? You should use the spreadsheet. You should get a spreadsheet like this. And you should like do some experiments and say, oh, OK, like, what if I just hardcoded this to false all the time.
Hamel: I think this confusion matrix may not necessary, might confuse people.
Shreya: So it’s gonna confuse people. It always does. You can’t teach the whole course in a 1.5 hour thing, I think, which is cut our losses.
Hamel: Yeah, yeah, OK, so, you know, that’s kind of, there’s a lot of things that we didn’t cover here. One key thing is how to split your data set so you’re not overfitting your judge and you’re not inadvertently cheating. That’s a lot, that’s way too much to get into in a one-hour podcast, there’s no way we covered that. But just know that, OK, there’s a lot of nuance here and how you do this correctly, how you build the judge, how you get confidence. There’s ways to calculate your metrics, use this TPR TNR to like calculate what, you know, your real accuracy is, that’s, you know, we haven’t gone into that.
What We Didn’t Cover (52:38)
Hamel: There’s a lot of nuance on like, OK, how do you analyze agents, like how do you, you know, if you have lots of steps and lots and lots of handoffs, how do you tame that and how do you do analysis of that to catch those errors? Another thing we teach is how do you do analysis of retrieval. So retrieval is kind of an Achilles’ heel of a lot of AI systems. And so a lot of times you have to kind of dive deep and diagnose what’s going on with your retrieval step in your RAG. And so there’s a host of metrics and analysis you might want to do there.
So, there’s a lot of things that we didn’t cover here, but the reason that we gave you a taste of error analysis, because error analysis is the step that most people skip in evals, and it’s rarely talked about. And it’s the thing that’s gonna give you extreme leverage as a PM and you can get there just with counting.
I hope that I’ve convinced you by using the spreadsheet that it’s within your reach. And you know, I don’t wanna discourage you from using spreadsheets either. Like, feel free to use whatever you’re comfortable with.
How to Improve Your System (54:01)
Aakash: So where do you go from here once you have this initial set of metrics? How do you go about improving once you have created your initial evals?
Hamel: Right. So, let’s say like this handoff error eval that we created, what you can do is you can, now you have a judge, an LLM judge that you like, you feel good enough about, it’s accurate enough. You can use it to score a large sample of all your production traces. And now you can find, you can learn, first of all, you can learn more about what is going wrong in those situations. But secondly, you can iterate on this problem really fast. You can make some changes to your prompt. And you can calculate your error rate on these test cases that you curated. And kind of iterate really fast and say, OK, like this prompt is working, this prompt is not working, and you will have a suite of these evals, and you can test against all of them to see, OK, like, if you are iterating on this problem, are you inadvertently breaking something else? And you have some kind of way to a system that you can use to sort of be confident in what you’re doing rather than just guessing.
What a Holistic Eval Suite Looks Like (55:14)
Aakash: What does a holistic eval suite look like?
Shreya: It’s different for every application and it’s different for how high stakes the application is. Typically I’ll see like, you know, several code-based evals, especially in CI, maybe one or two LLM-based evals in CI, but not really. I do see some people like myself included, run LLM powered evaluations, like kind of like monitoring or online, like every week or so, I’ll sample some of my traces, run my LLM powered evaluators on them. And then kind of just look at the score, see if anything’s off or whatnot.
And often I’ll see every few weeks that like, oh, there’s this new distribution of data or this new cohort of people who are using the tool—like I build AI powered data processing tools. And so I’ll see like, oh, there’s different document types that have come in or a different set of contracts. I do this for a lot, so like there’s a new type of contract or a new type of document that’s come up, and then now I need to like think about it, right? So LLM powered evals, automated evals allow me to really quickly iterate on those. You don’t need 100 of them, like just a few is fine.
The Role of PMs vs Engineers (56:34)
Aakash: And what’s the role of PM and AI engineer and AI researcher and all of this, how are you working together? Where are the handoffs happening, like that quick iteration on the system prompt, who’s doing that?
Hamel: It’s a really good question, you know, it depends on the size of the team and the company and the roles. Sometimes these roles are being collapsed into one. In terms of, OK, so, Jacob Carter, the CEO and also engineer of this product, is the product manager and the AI engineer all in one, so, you know, he has a pretty good pulse on, like, hey, is this interaction good or not?
That’s not feasible all the time, you know, in other situations, you want the domain expert to be driving, especially the error analysis process, as much as possible. It might take some training in the beginning to get the right tools surfaced for the PM or get the PM able to access the data and might need some engineering, like kind of co—what’s the right word, pair, pair programming in a way or pairing. So pairing on error analysis just to feel comfortable, but you should try to have one person do the error analysis so it doesn’t become onerous. And usually, a product manager is pretty good at that because they have all of the domain expertise to actually judge if something is going wrong.
So, I would bias on the side of having the domain expert or the product manager do this error analysis. And as far as like writing the prompt is concerned, you do want to try to make it accessible for the product manager to write the prompt. So, what I’ve seen in a lot of tools is having a, like an admin view where a non-technical person can edit the prompt.
I actually have like a screenshot of that here. One thing that we talk about a lot in our course is you want to create your own tools to look at your traces, and so Nurture Boss actually created their own—vibe coded their own tool to look at traces, to remove all the friction of looking at traces, cause it’s so important. And so, you know, this is pretty simple, you see all the different channels, voice, email, text, chatbot, you can see like they hide the system prompt by default, you know, it’s like a very quick and dirty interface on doing this open coding and axial coding, and actually they have a step here that helps them automate the axial coding, to see like, hey, transfer handoff issues, tour scheduling, blah blah blah. So, you know, that’s worth noting, that’s something to think about is that’s how important error analysis is.
Exposing Prompts to Product Managers (59:42)
Hamel: So to get back to your question about OK, how might you surface the prompt to non-technical people. So this is an example where you might have an admin view, so this is like, you know, a different real estate agent, like, hey, you know, showing you the real estate listings, and you might have this admin mode where then you allow someone to fiddle with the prompt. And so like this prompt experimentation is really key. And so, having a way that people can interact with prompts is really helpful.
Now, a lot of tools have like prompt playgrounds. The only thing that’s limiting about most prompt playgrounds is they don’t have access to your code, you know, cause you might have various tool calls, you might have RAG, you might have these things. All your application code is not in those prompt playgrounds, and so that’s why a lot of teams that I see have these interfaces where you can edit the prompt directly in your tool and then play with it and redo it and whatever.
So, yeah, so I mean whenever possible you want to expose the prompt to the domain expert because the reason is it’s English, it’s made for the domain expert. It’s almost a tragedy to separate the prompt from the product manager because it’s English.
Common Mistakes People Make (1:01:10)
Aakash: What are other mistakes like separating the prompt from the product manager that people might be doing in this process that we walk through today that is unintentionally inhibiting them?
Hamel: The main thing that’s inhibiting people is not doing the error analysis. People wanna jump straight to, hey, let me take an off the shelf metric that a vendor gives you and just like create a score. People are very scared of this error analysis. They look at it and they’re like, oh, I don’t have time for this. But it doesn’t take that long at all. And it’s kind of this thing, it’s like a secret club. You do it just once, you will forever keep doing it. It’s just like getting over that hump of doing it the first time. It’s just extremely scary for people.
Shreya: Another common mistake is people will see this video, or I don’t know, they’ll realize, OK, it’s worth doing error analysis. But then they think it’s worth some human doing error analysis, not them, so they’ll just outsource it out, which again, huge pitfall, right? This is the error analysis is where you build your product, right? That’s where you build your moat. So if you’re giving it to someone else, then you kind of have no personal touch in your product.
Hamel: Yeah, do not outsource this to developers. And, you know, if you’re working on a coding app, yes, you can, the domain expert is the developer, but in most cases domain expert is not the developer. And a lot of people, a lot of companies are like, oh this AI stuff is like for engineering, like the whole thing is engineering, let me just shove it over there. They need to figure out whether it’s like good or not. That’s usually the wrong approach. It’s not in the engineering skill set.
Shreya: I think that’s another interesting thing about today’s day and age for PM, especially AIPM, but you can’t expect engineers to be able to do all of these things. The people that have been successful at this process, either very, very technical PMs or are engineers who are actually PMs. And they just think that they’re engineers and they don’t realize that they’re doing product work, right? So I hope people are kind of convinced that today’s day and age, you kind of have to have your product knowledge, put your product head on.
The Power of Error Analysis (1:03:47)
Hamel: This error analysis is so powerful that this is a video and we could put it in the show notes. This is Jacob Carter and he just recorded like a two-minute long conversation of how thrilled he was with error analysis. He’s actually so thrilled with it that he just—he thought this is the best thing that’s ever happened. And he got so much value out of it that, yeah, like he kind of stopped there, to begin with, and had so much work to do that he didn’t need to build evals right away, cause he just found so many things, as you can see right here in this picture, that, you know, he was able to fix.
So, he did eventually build evals, but, you know, starting here gives you a really good grounding and lets you work through issues and get to evals for things that make sense.
Aakash: Going back to the beginning, there’s so much hype about what you’re trying to sell that your AI feature does, but to actually deliver on that hype, you have to go through these errors, so that when people are experiencing it in production, they actually get the experience you intended. And this has been our master class in how to do that step by step.
Where to Learn More (1:05:05)
Aakash: If people want to learn more, where can they find you guys?
Hamel: So the URL for the course, you can go to evals.info and you can find the course there, yeah, or follow us on X. Our websites are on the internet, you know, if you just look us up, we’re there, but check out evals.info. I think we’ve really tried to put together as much information that we can to be freely accessible and available to folks. So take a look where you can dive in and I’m sure you will learn things along the way.
Aakash: Awesome, thank you guys so much for—
Hamel: I wanna clarify that—sorry—so we mentioned like, hey, you need to look at traces in production. So you might be wondering like what if your application is not in production, what do you do? What if you don’t have data? What if you don’t have traces? Where do you get them? One, try to recruit some friends. Try to dog food your own app, that’s the best thing. For whatever reason, if you’re not able to recruit friends, you’re not able to dog food your product, which would be kind of sad, but let’s say, you know, there could be valid reasons you’re not able to do that, you can generate synthetic inputs into your system, and there’s a way to do that correctly. It’s kind of what you’re doing is you’re pretending to be a user. You’re having an LLM simulate that at scale. That’s one of the things that we go into in our course as well. So there’s ways to bootstrap yourself, but you do need to look at data.
Aakash: Amazing. Thank you guys so much for being here.
Hamel: Cool, thanks for having me.
Aakash: I hope you enjoyed that episode. If you could take a moment to double check that you have followed on Apple and Spotify podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube, all these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production to get you better insights to stay ahead in your career.
Finally, do check out my bundle at bundle.aakashg.com to get access to 9 AI products for an entire year for free. This includes dovetail, Mobbin, linear, reforge, build, Descript, and many other amazing tools that will help you as an AI product manager or builder, succeed. I’ll see you in the next episode.