Check out the conversation on Apple, Spotify and YouTube.
Intro (0:00)
Aakash: What are we going to do today?
Hamel: So today we’re going to walk you through how to do evals step by step. A real live example on real data.
Aakash: Do we even need evals? I heard Claude Code doesn’t use evals.
Hamel: Oh my gosh, this is a crazy controversy that’s been going around. There’s way too much hype about AI. To build a really good AI feature, it’s not just a demo. You need to build something that goes to production. I consider AI evals the number one most important new skill for product managers.
Aakash: Where do people at Anthropic and OpenAI go to learn AI evals? It’s Hamel Husain and Shreya Shankar.
Shreya: This is what your AI agents are actually doing out there in production and that’s why looking at the traces is so important. We show so many demos in class where we just dump this trace into ChatGPT and we ask, was the assistant correct? And then ChatGPT will say, yeah absolutely. But it will miss all of this nuance.
Aakash: Do I need an AI observability tool? I already am paying for whatever Datadog or whatever APM tool I already have. What is the difference?
Hamel: You don’t necessarily need a tool. It’s sometimes good to start with one, but if you want to, you can log to a CSV file, JSON file, text file, whatever you feel comfortable with.
Aakash: What are other mistakes that people might be doing in this process that we walk through today that is unintentionally inhibiting them?
Hamel: The main thing that’s inhibiting people is not doing the error analysis.
Why Every AI Product Needs Evals (2:09)
Aakash: Hamel, Shreya, welcome to the podcast.
Hamel: Thank you for having us again.
Shreya: Yeah, super excited.
Aakash: What are we going to do today?
Hamel: So today we’re going to take you step by step on how to do application-specific evals.
Aakash: Do we even need evals? I heard Claude Code doesn’t use evals.
Hamel: Oh my gosh, this is a crazy controversy that’s been going around. Absolutely, everyone needs evals. Some people are less rigorous about it because perhaps there’s somebody else who’s done evals for them upstream. For example, in coding agents, you know that people who are training the models are testing on a bunch of code. So maybe you can easily build a coding application based on religiously dog-fooding your outputs. But for most applications that are not just naive applications of foundation models, such as what you’re building, you’re going to need some form of evals.
Shreya: Couldn’t agree more. Evals, it’s about actually improving your product. Maybe you’re doing that through dog-fooding or maybe you’re doing it through the systematic process that we’re about to walk you through, but you have to do them.
Real Example: Nurture Boss Case Study (3:11)
Hamel: So let’s get started. Today I’m going to walk you through how to go about evals using a real company that I worked with, Nurture Boss. Nurture Boss has been very generous in allowing me to use some of their anonymized data as a teaching example.
What is Nurture Boss? Nurture Boss is a tool that allows property managers who are managing apartment complexes deal with things like tenant interaction, marketing, and sales. You can see their website here, nurtureboss.io, and you can get a feel for some of their product. It’s a mobile app, you can have a mobile app, you can embed it on the website. Here’s an example interaction: “Do you have any two bedrooms available?” and the Nurture Boss application is interacting with the tenant for you, helping to show listings, schedule appointments, so on and so forth. All the different activities that you might be engaged in as a property manager, their application is helping you manage that with the assistance of AI.
It’s a really good example because it incorporates all the messiness of a real-world AI application. There are tool calls, there’s RAG, multi-turn conversations. There are even multiple channels you can interact with the application through: voice, text message, or chatbot. So it’s a lot of different messiness of the real world. This is not a simplified example. This is something that you will encounter in the real world. Your application might have these complexities.
So how do you go about thinking about evals? When I started working with Nurture Boss, they had something initially that worked, but they really wanted to know, number one, how do we figure out what’s going wrong, and number two, how do we improve the application systematically beyond just doing vibe checks? They already did vibe checks. They were using their own application, they had some design partners and some initial customers, but they wanted to move beyond that and make it really good.
Starting with Observability (5:26)
Hamel: The first thing you want to start with is some kind of observability. The Nurture Boss application instrumented their code and they captured their traces. Let me show you what a trace looks like. This is an observability platform called Braintrust, and it doesn’t really matter what you use. There are a lot of popular ones out there. Braintrust is one. LangSmith is another one that’s popular. Arize is another one. It really doesn’t matter which one you use. The reason I’m showing you one is so that you get a feel for what traces might look like and also learn what a trace is.
Aakash: Before we even get there, some people might be wondering, do I need an AI observability tool? I already am paying for whatever Datadog or whatever APM tool I already have. What is the difference between those tools?
Hamel: You don’t necessarily need a tool. It’s sometimes good to start with one, but if you want to, you can log to a CSV file, JSON file, text file, whatever you feel comfortable with. The reason I have it pulled up right here is so we can read it together and have something to look at. But if you’re using Datadog, feel free to log things to Datadog to begin with. The most important thing that you’re going to want to have is to take notes on your traces, and we’ll show you why in a second.
One of the things that Shreya and I teach is to actually vibe code your own trace viewer. We’ll talk about that in a moment, and I can show you, Nurture Boss did vibe code their own trace viewer eventually. They didn’t end up using this. Sometimes to get started it is handy to use an observability platform, but sometimes it’s not. Depending on what you want, maybe you already have one, feel free to use that. The key is do the simplest thing you can think of. Get started.
Okay, so here’s a trace, and you can see the trace just logs all the different turns and all the information that is shown to the LLM. What you have here is a system prompt: “You are an AI assistant working as a leasing team member at Harmony,” which is the name of a fictitious apartment complex. “Your primary role is to respond to text messages from both residents and prospective residents.” So it’s very interesting, people will be text messaging this application. “You’re engaged with the customer to answer questions, book tours, and drive applications.”
There’s a whole host of different rules about how to respond to the customer. You have some rules: how to interact with them, determine if the inquiry is from a resident or prospective resident. I won’t read all of these for you. There’s some property-specific information here like URLs. This is anonymized. Obviously, the URL is not acme.com. But you get the idea. This is basically the system prompt.
You see the first user message. There’s some kind of logging error: “Don’t know who this is or what apartment you’re from.” So you can see this is real-world messy logging. The first question is: “I need a one-bedroom with the bathroom not connected in floor plans.” Then there’s a tool call of some kind where we’re getting the individual’s information, we’re getting something about the availability, and the tool is returning a list of apartments. You can see these are a list of apartments. Then we have the assistant responding with these apartments.
It’s saying, okay, we have these three apartments with a link to this floor plans page, so on and so forth. The user is responding, it’s a text message. The user is saying something about being sick and not being able to book tours, and something about wanting a bathroom not connected to the room. The assistant says, “I’ll check for one-bedroom apartments. So the bathroom is not connected.” Thank you. And you’re welcome.
So a lot of things kind of went wrong here. One is, what’s going on with “I want a bathroom connected to the room”? It just said, “I’ll check on that,” but then it didn’t do anything. We would hope that it would actually do something, or if it’s not able to do it, hand off to a human. So that’s funny.
I happen to know that this response up here is in markdown. This is a markdown response and this is a text message. So it’s going to be rendered a bit weird in a text message because text messages don’t have markdown. This is a bit problematic too with the bold and everything. It’s going to come across in a weird way with asterisks and stuff.
Aakash: Yeah, it’s going to have asterisks and it’s going to have square brackets and stuff like that.
[Sponsor break: Maven, Vanta]
Analyzing Traces (13:05)
Shreya: One more thing is actually in the first message the user said they wanted the bathroom and bedroom disconnected.
Aakash: Yeah, “I need one bedroom with a bathroom not connected.” And then the assistant’s first message was, “Here are some bedrooms with bathrooms connected.”
Hamel: Oh yes, there you go. That’s a good observation. So it actually didn’t really help the user. The user just kind of gently reminded them, “Hey, I do want a bathroom not connected to the room.” This is very messy. You could see there’s misspellings. I could almost not understand what the person was asking.
Shreya: The “now” should be a “not.” “I do not want a bathroom connected to the room.”
Aakash: This is awesome. Guys, if you are PMs listening, this is what your AI agents are actually doing out there in production a lot of the time. Your demo is one thing, it goes well. But then when it goes out in production, there’s all this hairiness. That’s why looking at the traces is so important.
Shreya: Yes. This is exactly why you don’t want to have generic metrics. If you try to put helpfulness score, conciseness score, whatever in here, or you try to have AI look through your traces, it’s not going to catch stuff like this very well at all because there’s a lot of context that you have as a PM, and a lot of things where you have taste that you need to reflect on and say, “Hey, this is not a good experience from a product perspective.”
The language model is not going to know that because it hasn’t been able to read your mind. This happens all the time. We show so many demos in class where we just dump this trace into ChatGPT or Claude, and we ask, “Was the assistant correct?” And then ChatGPT will say, “Yeah, absolutely.” It sounds correct. But it will miss all of this nuance that Hamel and I and Aakash have been mentioning, because we actually put our product hat on and thought about the user experience a little bit.
Aakash: Let’s see what ChatGPT does. Oh, look at that. So I found that one. It figured out that we didn’t get the connected bathroom. But this is hilarious. It says it doesn’t filter by bathroom configuration. The interesting thing is, who knows if that’s a filter that the tool provides. Yeah. The assistant cherry-picked three examples.
Hamel: I mean, maybe that’s fine for us, right? Nobody wants to see every single one-bedroom. I don’t want to see a text message of every single apartment. I actually only want to see a couple.
Aakash: Yep.
Hamel: So ChatGPT might help you a little bit, but you ultimately need to put your human touch on top of this and make sure that it’s correct. It won’t know about the markdown, you know. It didn’t catch that. You can change the format of this rendering somehow to show you the raw. This is actually all rendered as markdown but, you know, since this is a text message, ChatGPT is not going to know that. There’s other examples we’ll see where you need to have a keen eye about what’s going on in the product and bring your whole product knowledge to bear. If you try, it can miss a lot of things.
So what you can do from here is you need to write a note. I’m going to go into review mode. Let me find that trace again. This is also why we encourage people to build their own tools.
Okay, here we go. Notes. I would put a note here. Some issues here would be: told user that it would check on bathrooms but didn’t do it. Also did not follow user instructions, and rendered markdown in a text message.
What you really want to do is, this can sound very tedious. It sounds like it’s very resource and time intensive, but it’s really not. You just scan the trace and if you’re familiar with the system prompt, you don’t have to read it. You’re not going to read every system prompt because it’s going to be the same really, unless you need to. But within about 30 seconds or so or less, you can scan this and say, “Hmm, okay.” You can get a sense of what is happening and you can write some notes. Perfection is not key. The key is see what’s going wrong in the trace and note what you see and move on. You don’t have to catch everything. Just catch the most important things.
Let’s go on to the next trace. Let me find one with an issue. Okay, here’s one. This is a new trace now. The user is asking, “Do you all have one-bedroom with study available? I saw it on the virtual tours.” There’s a tool call to get information and availability. And it says, “Yes, hi Priya. We currently have several one-bedroom apartments available, but none specifically listed with a study.” Okay, so it matches. She did ask for a study. So I gave her one-bedrooms instead.
Then the user asked, “Can you let me know when one with a study is available?” And the assistant says, “I currently don’t have specific information on the availability of a one-bedroom.” Okay. This is where you get frustrated, right? I asked the question, you just responded with some robotic, “I don’t have that.” This is a product failure. So you want to note that real quick. “Should have handed off to a human or had better lead nurturing.” There’s no pun intended with the word nurturing. That’s just what came to mind. Anything else that you think is wrong with this particular trace?
Shreya: I don’t think you have to get bogged down. It’s a good question to ask, but I would tell people, all right, like think of everything that comes to mind, break them down, move on. You want to get into kind of a flow state here. You can debate every trace endlessly and sometimes you see people get stuck in that. So try to avoid that. We got a problem. Next.
Hamel: Yeah, I agree with that. Move on. This one I already did, but that’s okay, we can do it again. Let me find the first user message.
Okay, this is the third trace we’re looking at. The user asks, “I’m in California, looking to relocate to Texas by March 15th.” “Booven, thanks for sharing. Since you’re planning to relocate, blah blah blah, I can help you explore available apartments if you’d like. We can also schedule a virtual tour.”
He’s like, “Yeah, that’s great.” And, “Thank you. Okay, I’ll arrange a virtual tour for you so you can explore the community. What’s your preferred date and time?” “Tomorrow is fine for me, 9:00 a.m.” “I can schedule your tour for you.” It schedules a tour. Then it says, “Your virtual tour is all set.” Looks good, right?
Actually, it didn’t go so well. The reason is because there’s no such thing as a virtual tour for this apartment. I don’t think there’s such a thing as virtual tours for most of their apartments that they have on the platform. So the AI called the tour scheduling tool, but it doesn’t do virtual tours. So the platform scheduled an in-person tour and maybe the user is confused, and it’s like, “Oh great, it’s going to be a virtual tour.” It’s kind of a little bit of a disaster.
Then the date they said was January 22nd, 2025. The user said, “Wait, today is the 22nd. Do you mean tomorrow? Because we can’t do it today.” It’s like, “Oh, no problem. Your virtual tour has been rescheduled for tomorrow.” But if you look, the schedule a tour tool was called again. So what this means is, hey, you just scheduled another tour. You didn’t reschedule anything. So now the person has two tours. Maybe that’s going to cause a problem for the apartment complex because now they have two tours booked which are going to be no-shows.
Then there are some other questions: “How can I go about getting a unit if I’m in California trying to relocate?” blah blah blah. It’s giving some responses that seem okay. So I have written down the two issues here: we don’t do rescheduling, and we don’t do virtual tours, only in-person tours. So I just wrote some notes.
The idea is you just keep doing this. You can do this quite fast. You do this for let’s say 100 or so traces and just write down what you see. Don’t try to get into root cause analysis. Don’t try to figure out what went wrong exactly. Just journal, observe freely. Just journal what you see going wrong. If anything. If nothing is wrong, you can just skip it. But when you do find something wrong, go ahead and write that down.
Aakash: Yep, exactly.
Error Analysis Introduction (24:55)
Hamel: So now you have what we call a bunch of open codes. This is the start of the most important part of evals, which is called error analysis. It’s something that’s very approachable to everyone, and it’s actually very important for product managers to be involved in this because a lot of times engineers don’t have the full context to know if this is good or bad.
What you end up having is a bunch of these notes. So I have a spreadsheet open right now with a collection of all the notes that I took. Let’s say I did 100 of these. I actually found 40 or so different errors. You might find a different number of errors if you’re doing it. Here’s a collection of these notes.
Up until this step, you’ve already learned quite a lot. If you’ve looked at 100 traces, you’re going to learn and you’re going to understand your system better than anyone else. You’re going to have a really deep understanding of what is wrong. You might also have a pretty good sense of what you need to work on next already, without doing any analysis. But it’s really good to do analysis of these notes that you took.
Axial Coding Explained (27:00)
Hamel: How do you do this analysis? The next step is you categorize these notes. The term for that is called axial coding. I’m going to show you how to do this in a spreadsheet.
One thing you can do is put them into ChatGPT or Claude. So what I did is I took the logs, I exported it from here. There’s an export button and I said, “Okay, download as CSV.” I downloaded it, put it in Claude, and I said, “Hey, there’s a metadata field which has a nested field called znote that contains all the open codes.” I used the word “open codes.” That’s a term of art that LLMs understand. These terms, “open coding” and “axial coding” — open coding is the writing of the notes. That’s actually a term well understood in the field of machine learning, but it’s also been around before machine learning. It’s been used in the social sciences. This kind of process of open coding, axial coding is a thing that LLMs understand.
I just say, “There’s a metadata field which has a nested field called znote that contains open codes for analysis of LLM logs that we are conducting. Please extract all the different open codes and then propose five to six categories that we can create axial codes from.”
Then it’ll go through and you can get these categories. So here are some categories like: capability limitations, misrepresentation. Some of these I don’t like because they’re a little bit too broad. They’re not actionable. I’m not 100% sure what that means, so I might look into it and rename it a bit. Human handoff issues, certainly some of that. That’s when you want to escalate to a human being or hand off to a human being, but it’s not doing it properly. Temporal contextual awareness — doesn’t know what the current date and time is. So there are some categories here.
You can refine this. What I like to do is take it to a spreadsheet. So I have some categories that I maybe have from ChatGPT and then I look at them and edit them. I have these categories that I edited a bit, and I said, “Okay, let me just collect these into a list.” That’s what this formula does, just collecting these list of categories into a list.
Shreya: I think one of the things that’s very interesting from looking at your Claude is a lot of those axial codes are very vague, right? Like quality or temporal issues. You kind of want to make sure your actual axial codes are not so vague, because imagine you’re giving them to somebody else to do some labeling with. Something like conversational flow issues might be a little bit better. Honestly, we could even make it a little bit more specific. But something like temporal issues — if Hamel told me to go label with “temporal issues” category, I wouldn’t even know what he means. I would want to say like, you know, “date formatting error” or something like that. So I think that’s another place where people get tripped up, which is just taking things out of the LLM as is and not really thinking about, okay, how do I refine that in a way that’s going to give me meaningful error categories?
[Sponsor break: Jira Product Discovery, Land PM Job, Pendo]
Counting Issues (32:40)
Hamel: Exactly right. And that’s a really important thing to pay attention to. That’s why if you reflect on the categories I have in this spreadsheet, they’re definitely better than the ones in the Claude. They’re different for a reason. It’s because I iterated on it a bit, and that’s an important principle. You never want to completely hand off the wheel to AI. You want to think about what it’s saying. Maybe it helps you to different degrees, but you want to see, okay, what are the categories here? You might want to go back and forth.
What I did here is I went to those notes which I have here. Every row is a different note. You can use AI. So I used AI: “Categorize the following note into one of the following categories.” This is a formula in a spreadsheet. So you can see the prompt. Basically classify each of these notes into one of those categories.
I went back and forth like, “Hmm, this category actually is not the greatest for this particular note.” I went and edited the note. I went back to this category field, maybe added one, deleted one, and fiddled with it till I was reasonably happy. Like, okay, this is a good set, is good enough.
One thing I should have added here is a “none of the above” category, which would have been better, but I’m showing you the simple, stupid version, which is “get started.” I don’t want to overcomplicate it, but that’s what you should do. None of the above is mainly a means to the end. The end is really having these categories, but sometimes you might have missed a category. So if you put “none of the above” here and then your AI does a classification and tells you “none of the above,” then you can go read those traces again and wonder, maybe there’s another category that I should add so I can classify those.
So you have these classifications. Now comes the powerful part where you will have real superpowers if you do this as a PM. You will go above and beyond and be armed with information that a lot of people usually are not armed with, and it’s counting. Now you can count these issues. You can just use a pivot table. That’s what I did here. Say, “Okay, how many times did I see this?” So now you have taken a world where it’s kind of messy and you don’t really know what is going on. You might not know what is going on. You know that there are some errors, and you have this paralysis of, “What do I work on? What do I fix next? What’s the most burning problem in my app?” And now you have some data in front of you. You’re having these conversational flow issues a lot, and this conversational flow issue actually is regarding situations where there are text messages.
I happen to know I can click on this, you can say, “Hey, yeah, it’s like disparate messaging, sent in-person tour link.” There are different — sometimes it’s about text messages, sometimes it’s just like it’s not being here. We can go back to the trace and look at that. It’s one of my favorite things about pivot tables. You can double-click. You can also make it hierarchical. I like to do this too. Sometimes I like to break down conversational flow into three different subcategories. Maybe some will be repeated messages, and some things will be the AI just should have handled this one particular thing better. I don’t know.
Shreya: I think this is kind of where the subjectivity and your product experience shines, right? It’s like you have to do this process in a way that enables you to make your product better based on the capabilities of your product or what your team can do. So if you can’t have virtual tours, then you can’t have virtual tours, that has to somehow be encoded in your system, right?
Hamel: Yeah, definitely. This could be made better. I didn’t try to make it perfect. But as Shreya points out, you can have subcategories which can help you refine what’s going on more.
You can take a look at this. You can say, “Okay, what do I think is most important?” Like, hey, okay, maybe the human handoff issue is not happening as much as the conversational flow issue, but let’s say you feel as a product manager that is a catastrophic error, and the magnitude of that problem, the impact of that problem is so high that I’m going to prioritize that as number one. But you have some data to back up that this is happening, and you have an idea of what’s happening. Now you have a reason to potentially write evals.
Now you’re not writing evals in the dark. Now you can write evals in response to actual problems that you are seeing, instead of like a hallucination score or some AI-generated something or other. You can motivate this in things that you want to fix.
Now you don’t have to write an eval for everything. There might be some of these things that might be easy to fix. For example, there’s this formatting error with output. An example might be using markdown in text messages. You might be able to just fix that. Maybe there’s no instruction in the prompt at all.
It depends what kind of eval you need to write. There are two kinds of evals. One is code-based, where you can test something without calling an LLM. So the formatting error with output, you might be able to use a code-based eval for that. Like, hey, do I see markdown elements in this output where there shouldn’t be markdown? In which case maybe you should write the eval because it’s not going to be expensive. Whereas with the LLM as a judge, something more subjective like, “Hey, you should be handing off to a human,” you might need an LLM for that. You might not be able to write an assertion in code. That’s a little bit more expensive of an eval. You have to have a judgment call: okay, is that something that’s trivial to fix? Maybe you found some dumb mistake that you made, go ahead and fix it. You don’t have to get caught up in evals. What you want to do is write an eval for something that you think you might want to iterate against.
I don’t know if there’s a better way to say that. I feel like that could have been straight. You think there’s a better way to —
Shreya: No, I think it makes a lot of sense, right? Like already as a PM, this is the secret sauce for your product. If you don’t do this process, you can’t put your own taste or your company’s taste into your product. Once you get to this point, that’s kind of when paths diverge. I think that’s what Hamel is trying to say. Maybe sometimes you figure out, okay, there are some errors that are higher priority than others, I’m going to go and fix those. Maybe you want to run these checks at larger scale. Maybe you’re Meta or you’re Google and you’re like, “I can’t make a decision based on 42 traces and I need a lot of buy-in, so I’m just going to build a team, do a bunch of evals, or I’m going to write automated evaluators to check this at scale.” Yeah, go for it. We’re not going to cover that today. Take our course if you’re interested in those techniques. But overall, I think it’s super incredible. Hamel started with zero. Right now we’re at a place where we know what the biggest failure modes in a sample of traces are, right? And most people don’t get to this point.
Hamel: Going further from there, let’s say you want to write an eval for the human handoff issue. Like, hey, you should be escalating to a human being, you should be handing off. It can be really useful to write an eval to help you flag the traces where that might be happening and help you iterate on that problem. So let’s go into how you would go about building that.
Building Your LLM Judge (42:26)
Hamel: Here is the prompt for an LLM judge. Now this is just a very basic prompt. It could be made better, but I just want to keep it simple. “You are scoring a leasing assistant to determine if there’s a handoff failure. Return only true or false.” That’s one thing that we teach. You want the LLM as a judge to produce a binary score. Shreya, do you want to talk about why?
Shreya: All right, I’ll try to give us a sick answer for this. The short answer is that people run into a lot of misalignment when they try to use a Likert or a range-based score. That’s because it’s very tedious to check that every single possible LLM output aligns with your preference. Now, when you do a binary score, you only have to check that “true” aligns with your trues and “false” aligns with your falses, right? It’s only two things you have to check, which makes the process of checking for alignment easier.
The other thing is, when you’re shipping products, you make binary decisions. Either this thing was bad or this thing was good. I should fix this or I should not fix this. It’s not like — even if you have a score of “this is 30% failing,” that gets turned into a binary decision of how you’re going to act on it or not. So that’s why we tell people, focus on a binary decision here. It’s easier to align, and ultimately your business decisions are yes or no decisions.
Hamel: Yeah, there were some people who were trying to do one through five scales and stuff, but it seems like LLMs are not very good at those types of numbered scales. So it’s much better to stay binary.
Shreya: Definitely. We need to bring you into our consulting office if you can tell people that.
Hamel: We have this prompt. We have a list of seven things where there is a failure. An example of one is the user asks to be sent to a human, but that’s ignored. Or there’s a policy that you should be transferred, but that’s not handled properly. There’s a sensitive issue, like billing disputes or legal issues, that’s not adhered to. Same-day walk-in or tour requests, you want to hand that off to a human. Things like that. Then we have some notes about when there’s not a handoff failure.
It’s important not to get bogged down in the prompt itself. So if you’re thinking to yourself, “Can you send me this prompt? Can I copy and paste this prompt?” you’re asking the wrong question because you need to — okay, how do you write this prompt? You want to try to describe the rubric of okay, what is a failure and what is not a failure. You can get LLM to help you bootstrap that, but you should try to edit it. The key is iterating. It’s not necessarily a recipe. You want to try to have examples. I didn’t put examples in here. So you want to have a section of maybe some examples. It’s not necessary to begin with. In a simple case you may not need to have examples. I’m just trying to give you the dumbest LLM message so you can get the concept. The idea is you’re going to write a prompt. In our class we do have a recipe that you can follow, but zooming out from that, it’s important to just iterate honestly. That’s what’s going to get you the furthest.
This prompt would be structured differently if it wasn’t in a spreadsheet also. I’m kind of begging it to return true or false. I wouldn’t have to beg it if I was using an API, for example. I could do something else. So this is the prompt. Then what you can do is, okay, I have my trace here. This is a different tab of the spreadsheet. I’m saying, “Assess this LLM trace according to these rules,” and I give it the prompt of my LLM judge. That’s all I’m doing.
Aakash: Nice. AI function built into Google Sheets is just running Gemini in the back end?
Hamel: Yeah. It’s okay. I wouldn’t say it’s amazing. It’s some kind of very fast model. It’s good to get started, to get a mental model of what’s going on. But I would be a little bit careful using this model for everything in real life because I’m not too sure about it. Don’t get lost in the sauce of what I’m doing. I’m trying to give you a mental model, but you might want to use a more powerful model potentially for LLMs as a judge.
We have two columns here, column G and column H. Column H is the score outputted by the LLM judge: true or false. Most people just stop here. They’re like, “Okay, here’s my LLM judge, I gave it a prompt, woohoo, we’re done. LLM judge says it’s good, so we’re good, right?”
Measuring the Judge (48:02)
Hamel: What ends up happening is stakeholders start to feel or observe that there is a dissonance between your evals and the product’s actual performance, and they can lose trust in the evals. They start to ask you questions like, “How do you know this metric? What is this metric?” And you tell them, “Okay, it’s an LLM judge.” “Well, how do you trust that?” A lot of people get stumped there. Like, “Well, that’s all we got.” You don’t want to do that.
What you want to do is measure the judge against your label. Remember when we were doing the axial coding, you actually have your own human labels. So for these various traces, if this issue existed or not, you can compute metrics. You can compute how good your LLM as a judge is.
In this spreadsheet I have three metrics: agreement, TPR, and TNR. Now agreement is the trap metric. The reason it’s the trap metric is that’s what you might gravitate towards. In the naive case, you might say, “Okay, we just measure the agreement between the judge and the human.” You don’t want to do that. The reason is, if this failure is only happening, let’s say, 10% of the time, you can have the dumbest judge in the world have 90% accuracy by just always predicting pass. In fact, your LLM judge can just be like equals “pass.” You can introduce a bug that doesn’t even call an LLM. It’ll be 90% accurate.
So you don’t want to do that. That could be very misleading. It can mislead you. What you want to do is measure two things: how good is your judge at catching errors that exist, and how good is your judge at — sorry, let me rephrase that. I always get… let Shreya explain this one so I can give myself a break.
Shreya: Sure. I mean, I think you’ve basically said most of it. The point is, okay, if you’re a product manager and somebody tells you, “You have high agreement with your judge,” or they got high agreement with the judge, be a little bit suspicious. Ask them, “Okay, what’s the alignment in the positives or the passes truths? What’s the alignment in the falses?” And make sure both of them are pretty high individually. If not, then you have to rework your LLM judge prompt.
If you’re confused about this, why isn’t it — if you’re not convinced intellectually somehow that, “Why can I just use agreement, Hamel? Why do I need to measure positives and negatives separately?” you should use the spreadsheet. You should get a spreadsheet like this and you should do some experiments and say, “Oh, okay.” Like, what if I just hardcoded this to false all the time? I think this confusion matrix maybe — necessary. It might confuse people.
Hamel: It’s going to confuse people. Always does.
Shreya: We can’t teach the whole course in a one-and-a-half-hour thing. I think we just cut our losses.
Hamel: Yeah. There are a lot of things that we didn’t cover here. One key thing is how to split your data set so you’re not overfitting your judge and you’re not inadvertently cheating. That’s way too much to get into in a one-hour podcast. There’s no way we cover that. But just know there’s a lot of nuance here. How you do this correctly, how you build the judge, how you get confidence. There are ways to calculate your metrics, use this TPR, TNR to calculate what your real accuracy is. We haven’t gone into that.
There’s a lot of nuance on how do you analyze agents. If you have lots of steps and lots and lots of handoffs, how do you tame that and how do you do analysis of that to catch those errors? Another thing we teach is how do you do analysis of retrieval? Retrieval is kind of an Achilles heel of a lot of AI systems. A lot of times you have to dive deep and diagnose what’s going on with your retrieval step in your RAG. There’s a host of metrics and analysis you might want to do there.
There are a lot of things that we didn’t cover here, but the reason we gave you a taste of error analysis is because error analysis is the step that most people skip in evals, and it’s rarely talked about. It’s the thing that’s going to give you extreme leverage as a PM, and you can get there just with counting. I hope that I’ve convinced you by using this spreadsheet that it is within your reach. I don’t want to discourage you from using spreadsheets either. Feel free to use whatever you’re comfortable with.
Aakash: Where do you go from here once you have this initial set of metrics? How do you go about improving once you have created your initial evals?
Hamel: So let’s say like this handoff error eval that we created. What you can do is, you have a judge, an LLM judge, that you like. You feel good enough about, this is accurate enough. You can use it to score a large sample of all your production traces. Now you can learn more about what is going wrong in those situations. But secondly, you can iterate on this problem really fast. You can make some changes to your prompt, and you can calculate your error rate on these test cases that you curated, and iterate really fast and say, “Okay, this prompt is working, this prompt is not working.” You will have a suite of these evals, and you can test against all of them to see, okay, if you are iterating on this problem, are you inadvertently breaking something else? You have some kind of system that you can use to be confident in what you’re doing rather than just guessing.
Aakash: What does a holistic, end-state eval suite look like? Shreya, you want to talk about that? How many evals do you usually have in your…
Shreya: Yeah, it’s different for every application, and it’s different for how high stakes the application is. Typically I’ll see several code-based evals, especially in CI. Maybe one or two LLM-based evals in CI, but not really. I do see some people, like myself included, run LLM-powered evaluations like kind of like monitoring or online. Every week or so I’ll sample some of my traces, run my LLM-powered evaluators on them, and then kind of just look at the score, see if anything’s off or whatnot.
Often I’ll see every few weeks that, “Oh, there’s this new distribution of data or this new cohort of people who are using the tool.” I build AI-powered data processing tools. So I’ll see, “Oh, there are different document types that have come in or a different set of contracts.” I do this for law a lot. There’s a new type of contract or a new type of document that’s come up. Now I need to think about it. So LLM-powered evals, automated evals, allow me to really quickly iterate on those. You don’t need a hundred of them. Just a few is fine.
PM vs AI Engineer Roles (56:38)
Aakash: What’s the role of the PM and AI engineer and AI researcher in all of this? How are you working together? Where are the handoffs happening? That quick iteration on the system prompt, who’s doing that?
Hamel: It’s a really good question. It depends on the size of the team and the company and the roles. Sometimes these roles are being collapsed into one. Jacob Carter, the CEO and also engineer of this product, is the product manager and the AI engineer all in one. So he has a pretty good pulse on, “Hey, is this interaction good or not?” That’s not feasible all the time.
In other situations, you want the domain expert to be driving especially the error analysis process as much as possible. It might take some training in the beginning to get the right tools surfaced for the PM, or get the PM able to access the data, and might need some engineering pair programming, in a way, or pairing on error analysis just to feel comfortable. But you should try to have one person do the analysis so it doesn’t become onerous. Usually a product manager is pretty good at that because they have all of the domain expertise to actually judge if something is going wrong. So I would bias on the side of having the domain expert or the product manager do this error analysis.
As far as writing the prompt is concerned, you do want to try to make it accessible for the project manager to write the prompt. What I’ve seen in a lot of tools is having an admin view where a non-technical person can edit the prompt. I actually have a screenshot of that here. One thing that we talk about a lot in our course is you want to create your own tools to look at your traces. Nurture Boss actually vibe-coded their own tool to look at traces, to remove all the friction of looking at traces because it’s so important.
This is pretty simple. You see all the different channels: voice, email, text, chatbot. You can see they hide the system prompt by default. It’s a very quick and dirty interface on doing this open coding and axial coding. Actually, they have a step here that helps them automate the axial coding. You see, “Hey, transfer handoff issues, tour scheduling,” blah blah blah. That’s worth noting. That’s something to think about. That’s how important error analysis is.
So to get back to your question about how might you surface the prompt to non-technical people: this is an example where you might have an admin view. So this is a different real estate agent, “Hey, showing you real estate listings.” You might have this admin mode where you allow someone to fiddle with the prompt. This prompt experimentation is really key. Having a way that people can interact with prompts is really helpful.
Now, a lot of tools have prompt playgrounds. The only thing that’s limiting about most prompt playgrounds is they don’t have access to your code, because you might have various tool calls, you might have RAG, you might have these things. All your application code is not in those prompt playgrounds. That’s why a lot of teams that I see have these interfaces where you can edit the prompt directly in your tool and play with it and redo it.
Whenever possible, you want to expose the prompt to the domain expert because it’s English. It’s made for the domain expert. It’s almost a tragedy to separate the prompt from the product manager because it’s English.
Common Mistakes to Avoid (1:01:29)
Aakash: What are other mistakes, like separating the prompt from the product manager, that people might be doing in this process that we walk through today that is unintentionally inhibiting them?
Hamel: The main thing that’s inhibiting people is not doing the error analysis. People want to jump straight to, “Hey, let me take an off-the-shelf metric that a vendor gives you and just create a score.” People are very scared of this error analysis. They look at it and they’re like, “Oh, I don’t have time for this.” But it doesn’t take that long at all. It’s kind of this thing, it’s like a secret club. When you do it just once, you will forever keep doing it. But just getting over that hump of doing it the first time is just extremely scary for people.
Shreya: Another common mistake is people will see this video, or I don’t know, they’ll realize, “Okay, it’s worth doing error analysis,” but then they think it’s worth some human doing error analysis, not them. So they’ll just outsource it out, which again, huge pitfall. The error analysis is where you build your product, right? That’s where you build your moat. If you’re giving it to someone else, then you have no personal touch in your product.
Hamel: Do not outsource this to developers. If you’re working on a coding app, yes, the domain expert is the developer. But in most cases the domain expert is not the developer. A lot of companies are like, “Oh, this AI stuff is for engineering. The whole thing is engineering. Let me just shove it over there. They need to figure out whether it’s good or not.” That’s usually the wrong approach. It’s not in engineering’s skill set.
Shreya: I think that’s another interesting thing about today’s day and age for PMs, especially AI PMs. You can’t expect engineers to be able to do all of these things. The people that have been successful at this process either are very, very technical PMs, or are engineers who are actually PMs and they just think that they’re engineers and they don’t realize that they’re doing product work, right? So I hope people are kind of convinced that today’s day and age, you kind of have to have your product knowledge, put your product hat on.
Hamel: This error analysis is so powerful. We can put a video in the show notes of Jacob Carter. We recorded a two-minute long conversation of how thrilled he was with error analysis. He’s actually so thrilled with it that he thought this is the best thing that’s ever happened. He got so much value out of it that he kind of stopped there to begin with and had so much work to do that he didn’t need to build evals right away, because he just found so many things that he was able to fix. So he did eventually build evals, but starting here gives you a really good grounding and lets you work through issues and get to evals for things that make sense.
Aakash: Going back to our beginning, there’s so much hype about what you’re trying to sell that your AI feature does, but to actually deliver on that hype, you have to go through these errors so that when people are experiencing it in production, they actually get the experience you intended. This has been our masterclass in how to do that step by step. If people want to learn more, where can they find you guys?
Hamel: The URL for the course, you can go to evals.info and you can find the course there.
Shreya: Or follow us on X. Our websites are on the internet. You know, if you just look us up, we’re there. But check out evals.info. We’ve really tried to put together as much information as we can to be freely accessible and available to folks. So take a look. You can dive in and I’m sure you will learn things along the way.
Aakash: Awesome. Thank you guys so much.
Hamel: I might want to clarify that. We mentioned, “Hey, you need to look at traces in production.” So you might be wondering, what if your application is not in production? What do you do? What if you don’t have data? What if you don’t have traces? Where do you get them? One, try to recruit some friends. Try to dog-food your own app. That’s the best thing. For whatever reason, if you’re not able to recruit friends, you’re not able to dog-food your product, which would be kind of sad — but let’s say there could be valid reasons you’re not able to do that — you can generate synthetic inputs into your system. There’s a way to do that correctly. What you’re doing is you’re pretending to be a user. You’re having an LLM simulate that at scale. That’s one of the things that we go into in our course as well. So there are ways to bootstrap yourself, but you do need to look at data.
Aakash: Amazing. Thank you guys so much for being here.
Hamel: Cool. Thanks for having us.
