Check out the conversation on Apple, Spotify, and YouTube.
Why PMs should care about interview homework (1:25)
Aakash: Sumant, why should PMs even care about interview homework?
Sumant: First of all, thanks for inviting me on this podcast, Aakash. Really looking forward to our conversation. Let me answer your question.
In the past, and even now, companies want to assess candidates. But because there’s an influx of way too many, there’s a shortfall of hands on deck to assess all of them. Pre-interview assignments, homework assignments or take-home assignments give companies an opportunity to put the candidate to work before they actually assess their qualities and skills in an interview.
These assignments allow a company to see whether the candidate has the right skill set in terms of technical capability, problem solving, even presentation, and to keep an eye out for typical pitfalls. Stuff most companies would have to train people on. They can test all of that right here, within the take-home assignment.
The new thing, especially in today’s scenario, is that AI has made it way too easy for people to conjure stuff up. So companies are tightening the noose around what kind of submissions they ask for, and their push is towards making the candidate think rather than just providing AI slop. For this new age, candidates have to become a lot smarter.
Personally, with the candidates I’ve coached, I always ask them this question. If I gave the same question to your younger cousin, would they be able to prepare the same kind of slides you’ve prepared? Or would you have done a better job? If the answer is even 50% yes, I tell them to go back and prepare again.
Aakash: I love that. You need to demonstrate the unique experience and expertise you’ve developed in your years as a PM, or whatever background qualifies you to be a PM. Not how well you can prompt AI.
Which companies ask for interview homework (3:35)
Aakash: So who actually asks for interview homework, and who doesn’t?
Sumant: That’s actually a great question. Many companies like OpenAI, Anthropic, even Razorpay, CrowdStrike, Retool and so on. These companies have a smaller pool of people, fewer hands on deck to take hundreds of interviews. Also companies that have specific processes and use cases where they want AI PMs to come in and demonstrate their abilities using AI products. They’re usually the first ones to push out take-home assignments.
Larger, established companies like Google, Meta and other FAANG companies, who want to establish themselves at the frontier of best hiring practices, usually don’t do this. At least for senior roles, they never really give take-home assignments. There is one exception, which I’ve mentioned as well. Their early career programs might have this round in some cases. But usually you don’t expect take-home assignments from those companies.
Aakash: So is it fair to say, based on the companies you’ve listed, that AI companies or more tech-native companies are more likely to give homework?
Sumant: You can say that. A lot of AI native companies want to assess the real AI skills of a candidate. A lot of people who were product managers earlier have become AI-enabled product managers, but they’re still writing themselves up as AI PMs. AI native companies want to differentiate between the two. That’s why a lot of them like to give out case studies, specifically to discern whether the person has real functional knowledge about building AI products.
Where homework shows up in the interview process (5:34)
Aakash: Okay. So where does this show up? Is it typically in the middle of the process, the final round, or before you even interview? What’s the most common setup?
Sumant: It actually happens throughout the process. There’s the pre-interview screening. This is one of the most common steps. A lot of companies push for this because it allows them to filter people before they spend time iterating on a candidate, having multiple internal discussions, and then bringing that candidate in. They’d rather give a pre-interview screening, and only if you qualify are you brought in for an interview.
It can also come later. For example, there can be a mid-loop deep dive. In maybe the third or fourth round, a technical or behavioral round is converted into a deep dive round, where they ask questions like why did you choose this particular product, what made you prioritize this feature, which metrics are you suggesting and why.
And it’s not unheard of to have a final round presentation, where the candidate’s depth and problem solving skills are tested live in front of a panel, based on the case study they submitted.
Aakash: This one in particular feels very common, even for senior roles. What I’ve experienced is that juniors tend to get it early, even as the first step. Some associate product builder programs and associate product manager programs use it as a pre-interview screen. And for executives, when I was a VP of product and interviewed for a couple of VP of product roles, my final round would be a presentation to the CEO and CPO. Is that what you’ve seen? Is that fair to characterize broadly, or is that just my experience?
Sumant: No, that’s absolutely fair. Like I said, it’s not unheard of that even senior folks are brought in front of a panel and asked about the more strategic aspects of their decision-making. Why exactly are you choosing to go with this setup? What is the AI economics behind this decision? Those are questions you do expect from a VP.
The five types of PM homework (8:02)
Aakash: Love it. We’ve both experienced a similar thing coaching people, so I think it’s fair to say that’s the universe out there. What types of homework do people get, if we put it into broad buckets?
Sumant: Sure thing. I’ve categorized five types, the more prominent ones. I’m sure there are some specific to analytics, some specific to just building prototypes and so on. But here’s how I’d broadly categorize them.
The first one has got to be the PRD. For product roles in AI, they’ll ask you to generate a PRD for a vague and ambiguous problem statement. Recruiters are typically looking for the candidate’s composure. Do they ask questions back? Do they state their assumptions properly? The same old standard practices regular PMs have followed. Just that this time, the assumptions will be a lot more around AI.
Same goes for strategy and competition questions. It can be competitive benchmarking, and based on that, suggestions of what to build, what to improve, what to integrate, or maybe what to buy.
There can be roadmap and prioritization cases.
There can be regular case studies and problems, where the focus is more on user empathy, problem solving, and the candidate’s judgment and taste.
And the new thing we’re seeing these days is AI prototyping. This is the latest one, and for builder PM roles it’s the most commonly sought-after skill. I think this is what we’ll see the most in the coming two years for builder PM roles.
How candidates use AI to tackle homework prompts (9:42)
Aakash: Okay. And what do the prompts look like? I was just helping somebody with Netflix homework, and the prompt said submit no more than two pages for their strategy memo. Is that common? What are the most common prompts within these buckets?
Sumant: There are a number of ways people attempt these case studies. But more often than not, the prompts can include a structured markdown, where they divide a long process into a context-aware, decomposed prompt.
I’ll take the example of a PRD. When people build out a product requirements document, there are multiple phases. It’s not like we can write a functional PRD in one night. It evolves through various phases. So if we go to any of these LLMs and directly ask them to build a PRD, sure, they’ll generate something. But will it be useful? No.
So what we typically do is suggest candidates go through the process, write it down step by step, and show that process to AI, so the LLM understands the context and how to break it down the same way. Then it goes through it in sequence. That’s one big god prompt the candidate will probably write.
And it doesn’t just stop at generation. It goes through multiple recursive iterations. You probably have a red team evaluating. You probably have sub-agents checking every step and marking it up, or providing feedback, and based on that evaluation a new generation is created, and so on. The final result is actually pretty good, because it has gone through so many filters.
We also suggest our candidates do a quick human check. Basically a human in the loop. We tell candidates to constantly keep a human in the loop, so AI doesn’t go astray too many times or hallucinate. And if it does, the human is there to guide it.
Aakash: Amazing. And with the rise of Pangram and AI writing detection, and Anthropic and Google Gemini watermarking their text with SynthID, you can expect OpenAI will probably follow. AI is just making itself known. So it seems like the better model for people doing homework is AI as a thought partner, not as a writer.
Sumant: Absolutely.
Good example 1 – Taking an LLM incident triage tool to production (12:25)
Aakash: What does good look like? Can you walk through some real examples, so we’re not just talking but can really see it?
Sumant: Absolutely. Let me take you through some examples. I’ve taken real snapshots of presentations, slides and documents that were submitted. The good and the bad examples both come from real slides, shared with consent. Every piece of data here is what was submitted. There’s no mention of the person or the company anywhere. If there was, it has been redacted.
The focus will be on understanding the good examples, and then I’ll show you some bad ones as well. This first one was about an LLM incident triage proof of concept. The candidate had to make a case for production architecture, compliance and so on. What he did was lay out the PoC scope versus the production scope himself.
Aakash: Can I interrupt you here? I feel like the question itself is kind of tricky. If somebody is seeing this and doesn’t know what it is, what exactly were they asking for? I think even there you can learn a lot.
Sumant: Absolutely. Think of it this way. This is an incident triage tool, and the proof of concept already existed. The candidate was expected to go through the proof of concept and then make a case for the production architecture, compliance, rollout and success metrics. To take the PoC from there to production, what are the various things they’d need to do? That’s what the candidate was supposed to suggest.
Aakash: So one of the most important slides they’d have is, what does the PoC currently do versus what’s the production scope.
Sumant: Absolutely. I’ll quickly take you through some of what the candidate wrote. The actual details are not as important as the logic behind why this works. I’ll talk about that in a bit.
For example, the candidate mentioned that in the PoC, the incident JSON comes in from static files. But in production we won’t have static files, because it just doesn’t make sense. It’s production. It has to be real time. An incident can’t keep waiting to be read from a static file. So in production it has to be real-time event ingestion, maybe via APIs. The other lines follow the same kind of logic.
Now, why this works. Even though the candidate mentioned places where there would be one LLM call per incident to categorize it, or suggestions taken from an AI, the candidate didn’t take the easy path. The easiest path, which a lot of people would probably suggest, is to let AI handle it end to end. Let the AI handle the incident, write the summary, suggest what its priority should be and what the next steps should be. That’s basically a throw-it-in-ChatGPT-and-get-a-response mindset.
But the candidate went one step further. Throughout his submission, he had hard rules set above the model. The AI’s role, like you said, Aakash, was only there for suggestions. It was only supposed to be the thought partner. There were always rules that dictated whether what the AI suggested crossed a threshold, and only then could the decision be taken forward. And even then, there was a human in the loop. The final decision was taken by a human. That shows maturity. That shows real AI build experience. And that’s why this presentation was shortlisted.
Similarly, he had a very honest table, showing component by component, including what’s still only mocked. He included all of that despite it being a risk, because it was untested. It was just a mock. Despite that, he showed his vision of how it would look in the future.
I’ll show you one other slide. He laid out the value propositions very simply, with the impact clearly stated. For example, the actionable recommendation I was talking about. He said it would give operators a verified starting point for remediation, instead of letting AI handle it end to end, which can mean a lot of dollar value lost if something gets hallucinated. The impact is that operators’ lives become a lot easier. And he stated those impact points in real numbers.
He even had a slide specifically on how his proposal followed all the compliance requirements, and he quoted the exact lines from the compliance rules his proposal was following. This level of depth, this level of clarity, and honestly this level of transparency in presenting what you have in mind is quite rare. It shows maturity, and that’s one of the biggest reasons this presentation cut through. Makes sense?
Aakash: Yeah. It’s almost like you’re thinking about yourself versus the other three or four people doing the homework, and asking what am I demonstrating that the others may not. And maturity about what AI can and can’t do becomes a very powerful vector. It feels like this person really triple-clicked on that. Another thing this person focused on is they didn’t build super fancy slides. They focused more on the quality of the content.
Sumant: Absolutely.
The 20% most candidates miss – evals and economics (18:41)
Sumant: Let me demonstrate one more. But first, a summary of the previous slides. The whole presentation worked really well because of the systems thinking. The candidate showed audit trails, governance, a phased rollout and so on. They showed their experience in a regulated enterprise.
But what separates the regular good from the elite is this. When most candidates show their AI builds or ideas, they push some big top-line accuracy or precision numbers. But there’s no real mention of eval methodology or any cost model. Because of that, a lot of good submissions, good ideas and good content don’t make the cut. They missed one of the most important things.
So this is where most candidates land, just 80% of the way. That 20% is evals and the economics behind it, and that’s what many candidates either forget or miss completely. This deck actually showed that. That’s one big reason this guy got shortlisted.
Aakash: So what I’m taking away is, if we’re building an AI feature, let’s talk about evals. And for pretty much any feature, let’s connect it back to business impact.
Sumant: Sounds good.
Good example 2 – Unifying a portfolio of five data products (20:17)
Sumant: Let me show you one more example. The candidate applied to an AI data company, and they wanted him to unify a sprawling portfolio of five separate data products. These were products they already had, and they had a bunch of smaller products in their kitty as well. They gave him these five and asked him to propose new data insights. They gave him some sample tables, some structures and so on. The expectation was that the candidate would figure out what to do with it himself.
So he went through the huge amount of data they’d shared, did some research on competitors, and actually went and talked to some of his contacts to understand why users use certain types of data products. What do most companies, especially in the insurance sector, use that data for? Mostly reporting and template insights. They have separate teams doing just this. Then there’s a larger team doing analytics, which needs data exploration tools. And then there’s the tech team, which simply wants a dump of the data. So based on the users, this person segmented the list of products.
There were other ways as well. I remember, because I was the one who helped him through this, there were at least two more ways we could have sliced these products into buckets. We could have simply gone with free tier products and paid tier products. Instead we chose this, because he found the way users use a product to be a better way of classifying them. Not just today, but looking into the future.
It was his vision that later on, when we make many of these products AI native, say AI native exploration or AI native insight building, and as these products scale and newer products get built internally, like audit, even they’ll fit within one of these buckets.
That was the level of envisioning this person was able to do, and that’s not something any LLM can help you do directly. This is something that either you have or you don’t. That was one big reason his presentation got through the cut.
And he didn’t stop there. He laid out all the rationale behind this structure and even the trade-offs. He even mentioned the opportunity cost, which is very rare these days. A lot of candidates never really talk about opportunity cost as a trade-off.
How to get deep user understanding for a homework assignment (23:25)
Aakash: So the first thing you need to do is really have a deep understanding of the users before you go and start working on a solution. How did he get that deep understanding of how users actually experience the products? Even with the Netflix example I was helping somebody with, the user of that product is studios who make movies. But neither I nor the candidate had made movies before. So how do you really get inside their heads in a homework assignment?
Sumant: Fair enough. In this candidate’s case, he was from that same industry. He had experience building similar data products in the past, and he was from the insurance sector. He had about three or four years of experience working there.
So he understood how the typical insurance market works. You have a platform, you have partners, the TPAs, then you have employers who use these services, and the employees’ money is what’s actually invested in getting insurance. He already knew this. He had a lot of contacts, especially with partners. He used to work at a platform earlier, so he knew this value chain. All he had to do was find contacts who were employers and some employees. Employees are usually the easiest. Based on that, he was able to get insights. Real insights. It’s always better to get it straight from the horse’s mouth, isn’t it?
On top of that, he supplemented the insights he got directly with secondary research too. I have to give credit where it’s due. He uses a very interesting approach where he doesn’t depend on just one deep research run. He’ll have Perplexity doing deep research, Gemini deep research running at the same time, and probably a couple of agents on a different platform doing deep research as well. At the end, he accumulates the whole context and gets it summarized. So if one agent missed specific points, another one didn’t. Because he has everything accumulated, the evidence is more real. There are multiple signals coming in from various researchers.
That way he becomes extra sure that what’s being fed into his research is backed by multiple sources. It reduces the time he has to spend figuring out what was hallucinated, what’s real and what’s not. So his way of researching saves him time and makes it more accurate. More importantly, the signals he got from secondary research were quite coherent with what he found in his primary research. That’s how he went about it.
Should you take on every homework assignment? (26:28)
Aakash: Makes sense. So it sounds like this person succeeded partly because they had the domain experience. Is there a case to be made that you shouldn’t always do every homework assignment, because these can take a lot of time? Maybe you should only do the ones where you can actually differentiate yourself and have experience? Or should you take on every homework assignment?
Sumant: I have a pretty subjective take on this, because it applied to me as well. Back in my university days, I faced the same choice. We were applying to a lot of companies, and I made a call that I’d pick specific case studies and assignments where I had the best chance of getting through. I didn’t want to miss out on an opportunity that looked custom-built for me.
For the others, I’d make a call depending on how good my chances were. If I know there are candidates out there with better experience than me, who are a better fit for the role, and who will most probably do a better job on this case study because they have the right contacts and I don’t, then it’s probably in my best interest to save that time and effort and not incur the opportunity cost.
Aakash: What I’m learning from both these presentations is that the person themselves could bring some sort of alpha to the question being asked. So there’s a homework selection meta game you need to play. You need to ask, can I really even get this role? If so, then I’ll spend the time on this homework.
Sumant: The caveat being, if you push yourself for something, you go all in. It’s not a half-measure scenario. You’re not taking up other case studies so that you save time for the one you really want to go for. That’s what I did, and that’s what I recommend to all my students too.
Aakash: Okay, makes sense.
Insights backed by real data beat formatting (28:33)
Sumant: Now, when he submitted this, there was one more slide I’ll show you. This was the slide where he was supposed to show an insight that could be extracted from the tables they’d already provided. It’s quite text-heavy, because he knew he had to put in a lot of content. It would also be read by an AI at the other end, so he was okay with the format being like this.
The larger reason this got selected is that every insight here isn’t surface level. It’s not vague and ambiguous, or dependent on what we could build later. It’s actually built on the data that was already shared. These insights are extracted from real tables and columns in the initial data set. So the candidate is saying, I can use the data you’ve provided and bring out insights that probably aren’t even there right now. There were at least three more insights like this. This just shows that if a candidate wants to show depth, with a bit more transparency, this is a way to do it.
Aakash: Makes sense. So it sounds like an overriding lesson from both of these decks is that quality of insight matters more than formatting.
Sumant: Absolutely correct. After all, it’s not unheard of for companies to use AI to go through a lot of the assignments being submitted, and only the ones that get through the first AI pass reach human review.
So why did his presentations work? Because every claim he made, every insight he suggested, was backed by real data. Even the stakeholders he mapped each had their risks allocated. That’s essentially the same muscle as setting autonomy for an AI agent. What each agent can do, where each one is at risk, how you handle those contingencies.
And notice, in the strongest AI PM homework we just saw in these slides, there was barely any real mention of AI at all. Even though the insights mentioned that they’d be generated with generative AI, evaluating patterns and telling you what the right target would be, he didn’t go deep into talking about models. Not at all. The rest of his presentation also talked only about his ideas and insights.
Should you go beyond what the prompt asks for? (31:11)
Sumant: And that brings me to a very important point about constraints. A lot of the time, people like to go beyond the suggested number of slides or what’s been asked for. Candidates feel it’s a better idea to demonstrate that they can do more. What do you think, Aakash? Should candidates do that? We’ll discuss this in detail a bit later, but I want you to have an opinion on it.
Aakash: I don’t know, it feels like you’re setting me up to be wrong. But I think if you’re disciplined enough not to use AI, and it doesn’t smell like AI at all… In the Netflix example, they said give us a two-page document. So what we did is we had links in the two-page document. Here’s a link to the prototype. Here’s a link to our data study in case you want to learn more. And we just didn’t use AI throughout. But I’m curious what your take is.
Sumant: That’s actually very much aligned with what I have in mind, but we’ll talk about that later.
Bad example 1 – An MVP that tries to do everything (32:22)
Sumant: Now that I’ve shown you some good examples and why they work, let me show you some bad ones. I’m calling them bad examples, but remember, even one of these got shortlisted.
So this is the first one. The ask was to create an MVP for an app that revolutionizes the in-person live music experience. That was the broad idea. This is an actual slide this person submitted. If you notice, it shows the features of the MVP, and they’re all over the place.
I don’t know about you, Aakash, but I personally don’t feel this app, and the way it planned to revolutionize the in-person live music experience, really required all of these. I don’t think this is the right definition of an MVP. It looks like whatever came to mind, nobody said no to it. Everything just accumulated and went into the MVP.
When recruiters see this going all over the place, not really testing the one core idea that solves the main problem, that’s usually the red flag. We’d be one of the first people to say, okay, this guy, no. That was a big challenge with this one. Even though it’s written in extraneous detail and a lot of information has been provided, it misses the core idea of an MVP.
There’s more. In one of his final slides, he had a very unique take on how to make money. He mentioned aggregating anonymized user data to be sold to vendors and event organizers. Again, I don’t know about you, but I don’t believe selling user data is a great idea to mention at all, especially in a case study submission.
The funny thing is, this deck, back in ’23 when it was submitted, actually got through. It was a top company. Again, it was only for an APM role, so I guess they were just trying to give this person a chance. But today this would probably not even survive the screening round. The homework round itself would have at least three or four red flags, big P0 red flags, attached to it. And the moment it gets to more than two red flags, the candidate is out. So, a perfect bad example. What’s your take on this, Aakash?
Aakash: It sounds like you need to make sure you’re demonstrating that you’re making tough choices. Stick to the scope of the question. And if you’re going to propose something risky like selling user data, at least put the risks or the flip side on there. Hey, there could be some brand reputation risk. We need to examine whether this is standard practice, whether it can be anonymized. This might come up as something the public doesn’t like, so we need to understand whether it’s actually permissible. If they’d put that, it would have been better.
Sumant: Absolutely correct. That’s the point I wanted to highlight. There might have been examples where people got through even with this, but that doesn’t mean it’ll work today. Today, with AI doing first-round screenings, catching these becomes a lot easier. There isn’t even limited human oversight at that stage. AI is going to flag it, and the moment humans see it, you’re out.
Bad example 2 – AI features with no evals or economics (36:15)
Sumant: I’ll show you one more bad example. This is a whole different presentation, on a different topic, from a different candidate. This candidate was supposed to do a basic competitive analysis for a top tech platform and then suggest features to beat the competition. A very standard exercise a lot of PM or PM2 candidates get for homework.
What this person did was name two specific AI-powered features, very clearly. He named their pricing and their models and put them on the roadmap. But nowhere in the 14 slides did he mention anything on the specification front. What exactly is the data feeding them? What accuracy must these AI products actually have? What’s the cost per request? What will the feature economics look like? Will it actually hit any success metrics? Are there even particular success metrics for this? Nothing at all.
Essentially it became a quick advertisement for AI products, without any real depth of thought about what those products would actually be like, or whether they’re good propositions for us.
And it was made worse because I’d say he committed a blunder. He said he’d add this to the priority queue as an initiative with a realistic delivery in under six months. He mentioned everything. Telemetry, heuristics, fine-tuning, everything. But if you ask, is there an eval plan behind this? Nothing. No mention at all. In fact, even this was written with bare minimum detail.
Now here’s the question. If you’re not mentioning evals, if you’re not even including them as part of your realistic delivery, aren’t we going blindsided into this game? What if your AI is actually not that great? What if you’re just shooting blind? That’s the part that breaks the case for product managers, and that was the problem with this person’s presentation.
Unfortunately, this isn’t even the first time I’ve seen this. There are other cases, which I haven’t quoted in my presentation, where candidates made similar mistakes. Like I told you earlier, evals and economics are the two Es most candidates miss, because there’s just way too much excitement about showing what an AI can do. Who wants to care about the boring numbers, right?
Aakash: It almost feels like executives a couple of years ago who just wanted to sprinkle AI on top of everything. You can’t really do that in a PM homework assignment.
Sumant: Absolutely correct. That’s the problem. So these were some examples, both good and bad, all from real instances. Hopefully they added a lot of insight to our conversation.
Aakash: And some of these bad examples might even have been formatted better. So it really comes down to the insight.
How to split your time on a homework assignment (39:27)
Aakash: So what’s the repeatable process for a PM? If I’m given a homework assignment, how do I go about executing it? Let’s say I have 10 hours, or 6 hours. How do I use that time properly?
Sumant: I’m guessing 6 hours is more of a guideline, because many of these assignments might need more time than that. Some of them even want you to go out and research manually.
But if I have to give you a broad distribution, I’d say focus a lot more in the first couple of hours on understanding the problem statement in detail, without worrying about how to throw more AI into the mix. Yes, you’re building an AI product. It’s an assignment about an AI product. But the focus on problem solving is still going to be there. Ultimately, it’s the role of a product manager.
If you don’t have product sense, if you’re not getting to the crux of the problem, then no matter how many AI skills you have, no matter how good your understanding of evals is, it’s still not going to cut it. So focus the first few hours only on understanding the problem properly, understanding its nuances, figuring out why something is going wrong. Do as much research as possible. Take help from AI. Throw it all in.
The next couple of hours are where you push for execution. That was the planning phase. Now you start building out the structure of your response. What are the instances you’re going to quote? What are your assumptions? What are the open questions? What’s the outline of your answer? What’s each step that feeds into the next one?
One candidate example comes to mind. He followed the same structure, put together an outline of his presentation, and got it reviewed by me. I gave him the feedback that it just sounded too naive right now. He was jumping the gun. He’d done his research, but he was in too much of a rush to show the solutions he’d thought of.
I told him he had to go back and lay it out simply. This is the problem. This is the top priority problem, because you still have to prioritize. Then, based on that priority, show your solution, and even then mention the trade-offs. You don’t rush headlong into solutions. You mention the trade-offs and keep it flexible. You don’t say, with an arrogant tone, this is what it’s going to be and nothing else. In presentations, or in general when we’re proposing ideas, it’s always a better idea to have an air of modesty about it.
And finally, if you have time remaining, which hopefully you will these days using AI, go ahead and format it. If you’ve pushed for good content and have some time left, push for formatting. But make it readable. The first focus has to be readability. If it’s not readable, if you goof up outlines or alignment, it just looks unfiltered. Honestly, it looks uncouth. You’re a product manager. You’re expected to understand basic design. You don’t want to create a bad first impression, because this is your first impression when the slide reaches someone.
So have good enough formatting to keep it readable. Keep it in a format that’s readable even by an AI. Don’t overcomplicate it. Don’t make it too heavy with too many images. Links, yes, you can add. But other than that, keep it minimal if possible, and focus a lot more on readability and solving the problem step by step. That’s how I’d suggest a student distribute their time.
One caveat. There will be cases where even the allotted time isn’t sufficient, and many candidates send emails saying we need one more day because we were busy with something and couldn’t submit. It might have worked a few times in the past for you. But these days, recruiters know AI is available. My suggestion is, figure out a way to complete things on time.
When to build an AI prototype for your homework (44:15)
Aakash: So here we saw slides. A lot of people I’ve been talking to almost want to do an AI prototype for every homework assignment. Should you have an AI prototype in every assignment? And when does it even make sense to build a full-fledged app beyond a prototype?
Sumant: Makes sense. This brings us back to the question I was asking. Should candidates go ahead and do something that hasn’t been asked for? Is it a good idea to flaunt our prototyping skills?
So here’s the answer. If the role is a builder PM role, if the prompt you’ve been given in your take-home says you need to demonstrate or prototype it, or if the company’s own product is agentic, by all means go ahead and build a prototype. That’s as simple as it gets.
But if they’ve only asked for a PRD or just a document like a competitive benchmark, and you still go ahead and build an app they never really asked for, it has a tendency to backfire. It’ll be perceived as you can’t follow instructions. You’re spending time unnecessarily building something nobody asked for. Basically, it becomes a question of prioritization. They’d feel you’ll probably try to flaunt the same skills when they hire you, even though you have other things to do. That’s extra cost to the company, not extra credit.
Aakash: So don’t always just add it, unless the role or the company really calls for it.
Sumant: Absolutely. And even if you do build, maybe just build a thin slice, to showcase and supplement your ideas. It shouldn’t be a huge, broad, very elaborate mock. Show the evals, not just the demo part. The idea is, can you follow whatever instruction is being given and create something full of content and real value for the recruiters? After all, it’s part of the interview process. They’re testing your skills, and they have other questions to test your other skills. Don’t demonstrate everything they didn’t ask for in the same question. Stick to the prompt.
Should you stick exactly to the length limit? (46:52)
Aakash: And talking about sticking to the prompt, a lot of times they ask for just one page or two pages, and we want to add an appendix or more links. What’s your take on that? Should you stick exactly to the length requirement?
Sumant: Yes. If they asked for one page, that’s it. No negotiation on that one. When we were discussing this earlier, you mentioned it yourself. You submitted two pages. You didn’t add too many pages after that. You used a smart way. You had links in those pages and redirected people to them. But the submission was two pages only. That’s the point.
Following the constraint is also part of the test. Can you cut down the detail so it’s concise enough to fit in one page or two pages? Sometimes that’s what we as PMs need to do. We might not have all the space in the world on a small mobile phone screen, but we’ll have way too many features to accommodate. Can we do that? That’s what’s being tested. Especially as enterprise PMs, we live within these constraints almost all the time. Aakash, you’ve had your experience with enterprise products. What do you think?
Aakash: Yeah, I think so. Although I think the links might be a little bit of a hack. Say you’ve done all this work, like the first successful candidate. You actually went and talked to people in the four buckets, partners, ecosystems, customers, and you want to show that off. I think you can use links strategically. Maybe you recorded those calls. Maybe you went even further, put them into Dovetail and tagged the insights. Maybe you went further still and aggregated the insights and the quotes. If you really did all that work, I don’t think it hurts you to summarize the key point, the key takeaway, and then have a link. To me, that’s the one hack where maybe you can sometimes go beyond the constraint. And hopefully not too many people will think, oh, he’s just going outside the constraints.
Sumant: Absolutely agree. And that’s what this point is. You send one page if they ask for one, and you add a link for anyone who wants more. That’s it.
The biggest mistakes candidates make (49:12)
Aakash: So what are the biggest mistakes you see people make with homework?
Sumant: Oh, there are a lot, actually. I’ll jump straight in. Even the greatest PMs sometimes end up making these very common mistakes.
Like I mentioned earlier, sometimes there are very silly mistakes, like a typo on the first page of the deck itself. Interestingly, there’s a higher probability of missing things in the first half of anything you’re about to present than in the latter half. It happens because we typically have a bias towards the middle and the end. We feel we’ve already had a good look at the first part. It’s never the case. So this is a very common problem. Candidates end up polishing slide nine, where they’ve written the conclusion. But on the very first page, the intro slide where the presentation starts, they goof up a spelling or misalign some text. That’s a very bad impression to create, especially at the start. You build a perception in the recruiter’s mind that you lack attention to detail. Not the thing you want to create.
Next, answering the prompt they wish they’d got. It’s wishful thinking. Oh, I think the question means this, without going into the depth of what the question is really asking, or why a particular problem is being tackled. Instead the candidate just imagines what the problem must be asking. A lot of smart PMs fall into this too.
I remember one particular case. The candidate was asked to make a decision about adding a few features. He had to decide which of three suggested features would be the best bet for the company. They were supposed to do market analysis, pull together statistics and numbers, and make a calculated call based on that. That was the assignment. This person, in all their wisdom, perceived it as a benchmarking and parity matching exercise, and that’s all they did and all they submitted. They didn’t say which feature should actually be taken forward. They just compared the three features with what competitors were offering and showed a benchmark.
Similarly, especially with people who get AI to write things for them, you’ll see this a lot in their submissions. Sentences that say nothing, like leverage AI to revolutionize support. Revolutionize itself is right out there. That’s written by AI. Nobody talks like that, and yet you’ll see these terms. Whenever I see something like this, whenever I see em dashes, whenever I see those not-this-but-this constructions, those are absolute AI-isms, and they can be spotted from a mile away. Even with great PMs, you’ll sometimes see them think, ah, it’s just the conclusion, let’s get AI to summarize it. No. Please humanize it. Spend some time.
Then, missing numbers, or worse, adding numbers anywhere. Not adding a number where a number belongs, and the reverse, where you throw in numbers unnecessarily at a place where you’re drawing conclusions. Both are wrong. Understand where a number really belongs, and only put it there. Usually it’s about storytelling. A well-placed number supplements your story. It just makes sense to see that number at that point in the story. If you miss that, you’re telling the recruiter you’re not that good at telling a story. And for product managers, Aakash, you can attest to this, storytelling is a crucial skill. Very crucial, especially when you want to align your stakeholders. Not the perception you want to create in front of a recruiter.
And finally, one of the most important points, where I’ve personally seen a lot of people make mistakes. They skip the defense. The defense requires prep. You can expect to be grilled on the decisions in your presentation. How can you not have really solid points to back your claims? It’s not sufficient to just build a good presentation and crack that round. Remember that many times during the interviews you’ll be asked questions about it. You have to remember the statistics. You have to remember the format. You have to remember every decision that made you prioritize one feature above another, and the reasoning behind putting one slide before another. There are many more mistakes like this that a lot of candidates make, but these are the ones I’ve seen the most, so I thought I’d present them first.
Aakash: Preparing the defense is so important. And I think that’s where AI can actually be helpful. You can have AI read your document and the prompt and ask, what are the three biggest problems with this? What are the three questions they’re most likely to ask? Then practice, and maybe even check in with your recruiting buddies or coaches. This is where I feel coaching can be helpful too, doing those rounds of iteration. So if you’re given four days to build a homework assignment, try to finish it with one or two days left. Don’t rush until the end and email asking for another day. Actually finish a little early, so you can spend time on the defense. That’s very high ROI.
Sumant: Absolutely correct.
The system to succeed on PM homework (55:41)
Aakash: All right. So if we put it all together, everything people learned, what’s the system to succeed on homework?
Sumant: I’ll mention three broad points, and that will summarize the system.
First, be in a rush to understand the problem. Don’t be in a rush to just send AI slop.
Point number two, evaluate everything you’ve written and get it reviewed as many times as possible. First through AI and yourself, then with someone experienced, someone who has actually done hiring or evaluated these in the past. That will give you so much concrete evaluation and concrete feedback, and it adds a lot of value to your presentation.
And third, to succeed, focus a lot more on the content. Don’t worry too much about formatting. Just keep it readable. Don’t go for fancy templates. Focus on delivering value. Don’t try to throw in fancy AI keywords. A lot of people are getting into the habit of throwing in embeddings and this eval framework and whatnot. That’s all great, but focus more on the problem at hand and how you’re going to solve it. Don’t forget the evals. Don’t forget the metrics. Don’t forget the economic side of things. Mention those parts. You’re applying for an AI product role, after all. These are essential. Forget about the formatting, you can cover that later. Just laser focus on the value. That’s how you do it.
Where to find Sumant Malhotra (57:27)
Aakash: And that was a ton of value, all freely provided. Thank you so much for this deep dive. If people want to find you online, work with you or learn more, where should they go?
Sumant: Interestingly, Aakash, I’ve also started my YouTube journey. So they can find me on my YouTube channel. I’m also on LinkedIn, and I can be found on Topmate for one-on-one consultations and other plans.
Aakash: All right, guys. I’ve been watching Sumant’s content for a couple of years now. He seems like somebody who’s super dialed in, and if you’ve watched to the end, you’ve enjoyed his insights as much as I have. So go give him a follow and subscribe to his YouTube channel. We’ll be bringing you more PM job search content on this channel, so make sure you’ve subscribed here as well. Comment below with what you want me to cover next, and I’ll see you in the next episode.