Check out the conversation on Apple, Spotify and YouTube.
What loop engineering actually looks like (2:22)
Aakash: Let’s get started. How do PMs and product people use loops?
Tyler: It’s something we talk a lot about. I feel like AI more than most things has a lot of hype and not a lot of action. So I’d love to just show you what we do, and try to be really safe that I don’t show anything I’m not supposed to.
What I’ve got up here is the actual system I use every day to do loop engineering, context engineering, whatever you want to call it. We’ll talk a little bit about the differences. The system I’m in, for people that want to follow along, is called Herder, which lets you orchestrate agents and was built with the ability to essentially run itself.
So I’ve built this thing called orchestrate, which I can then tell to spin up another agent to research contractor needs in 2028. It actually knows in Herder, which you’ll see it do in a second here, how to spin up an agent, how to open a new pane on the side here, which I can move between.
You see I’m working on some different leadership stuff, hiring. These green dots tell me what’s done. This one’s ready for me to look at. Yellow is it’s moving. And this one I’ve already looked at. So to get to your question, this is really the pane of glass I use to manage multiple loops, which starts with getting an AI helping you.
Why terminal agents beat VS Code and Cursor (3:47)
Aakash: Why would I use Herder instead of VS Code or Cursor or Claude Code in the terminal?
Tyler: It’s a great question. I was actually pretty skeptical maybe a year ago about terminal-based coding agents, but they’ve gotten so powerful, and the systems around them like Herder have also been made in such a way that’s powerful, that it feels super native to work in the terminal where it can access what it needs to access.
I get that this isn’t as maybe sexy as VS Code, but for me I can just have a terminal open that’s an editor if I’m doing some coding. I can also in that terminal spin up an agent in Herder. I can even split panes and do all sorts of work. So it’s easier for me to have control over the agent loop cycle as opposed to having to rely on VS Code to release things at the pace the industry is moving.
And that’s one of the things that’s really fun about the community tools. Most of them are using AI to help build the tool. They move very fast and tend to evolve very quickly as AI improves.
What actually makes something an agent (4:48)
Aakash: We’re using this term agent. I feel like agent is used on so many things. What makes it an agent here versus just another Claude Code instance?
Tyler: Super solid question. For me, the difference is an agent is doing something on its own and making decisions, versus you consistently prompting it.
ChatGPT came out and most people are interacting with it at a level of question, answer, question, answer. I’ve heard people refer to this as almost like smarter Google search. Agentic is more you’re giving it a task with a way to verify itself and asking it to execute on that by itself as much as possible. So you’re taking yourself out of the loop more than you are with just a prompt.
That way you can actually scale yourself. Where I think prompting got hard is you’re constantly in the loop and you’re not really feeling the gains you get from agentic.
Using loops to prep for a podcast (5:41)
Aakash: Amazing. So what do we do next with this research contractor agent?
Tyler: It’s off. It’s doing its thing. It’s going to report back when it finishes. I believe it made it in, let’s see, right here, customer research. So if I click this, it’s doing research and drafting some thoughts for me, which isn’t really the point yet of this.
To show what I was thinking outside of Herder with loops, one of the things I was super interested in is, can I use loops or agents to help me prepare for this podcast? That’s the most meta thing I could possibly do. So I pointed it at a bunch of stuff we do at the company and asked it to build various demos we could run through to show different things.
What I’m doing actually is kicking off a predefined skill or agent I’ve built that knows the things I want to talk about. It will then walk us through some different examples.
One of the things I asked it to do, which makes it a little bit more of an agent loop, and you’ll see that some of these things I’m not actually passing, is a pre-flight check before I record. Am I showing anything sensitive? It’s saying yeah, you might be showing some sensitive stuff. I think we’re okay. Is the repo in a good state? I’m not actually in a sandbox, which I feel okay about. It’s got some fallbacks that are a little bit overkill. And then some manual things I’m supposed to do, like did we connect Riverside and all that other stuff.
I did this on purpose to show a few things. One, AI when you ask it to can be a great partner to help you do better. I wouldn’t have thought about half these things and it’s forcing me to check them. I did run this before and I decided I didn’t care about a sandbox. We’re okay for this podcast. But every time now, if I ever wanted to use this in the future, that loop is starting off with conditions I’ve told it are required for success.
Now it’s asking me, is this a rehearsal or is this actually a live run? And we’re live. So let’s say it’s live. Do we want to start at segment one or start where I finished my rehearsal? Let’s start from the beginning.
You can see, in my opinion, how this goes beyond prompting and into building an actual system to help me prepare or do work in a way that I feel good about and isn’t just AI slop.
What makes it a loop (7:47)
Aakash: You called it a loop specifically. What makes it a loop?
Tyler: We’ll see at the end here. One of the things it’ll do, assuming it all runs as planned, is towards the end the way you close that loop is you feed information back into the AI to make this whole system better.
I actually have a little graph we can run through in a minute. It’ll pull it up. But looping assumes that you learn, and it builds this flywheel. Any good loop has this end that says, hey AI, let’s review what I did. Let’s look at the log of all this back and forth, and how do we take that and improve it.
That’s really where the loop comes from. If you hear people talk about looping, it’s this idea of self-improving agents as opposed to a static skill. The skill could just sit here and be the same forever. Or we could take this entire experience and feed it back into the skill and have the AI improve it.
Version control and rolling back (8:45)
Aakash: What I get nervous about there is, is it going to save all the intermediate versions? Am I going to be able to easily revert back if there was some version of it that I liked before?
Tyler: For me, Git is the main way I handle that. Very normal for engineers, a little less normal for product people, though I think that’s changing. Git lets you version control these things. Actually one of the questions that popped up was saying, hey, your Git is unclean, what do you want to do? Oh, here it goes, it pulled it up.
I think that’s the right way to think about it, which is you can go back in time like Git. Maybe someone will make something more agentic than Git, and there are companies working on this. But with Git I can commit and move back in time. And those beeps you’re hearing is it telling me I’m done with something. So if you hear those through the podcast, that’s how I manage it.
One of the things I’d love to talk about is the importance of having your system set up so you can roll back and also adapt to new models. Because what we’re seeing is these loops you might have built, or these skills or agents or whatever you want to call them, different models perform differently. You might go to a better model like Opus 5 or Fable 5 and find that your skill gets worse. That’s kind of a paradox to work through, because sometimes you might have forced things that don’t need to be forced anymore.
Aakash: Yes, I felt that way about a lot of my Claude skills.
The anatomy of a loop (12:12)
Aakash: So what are we looking at here?
Tyler: When I was preparing for this, I started asking myself some of these questions. What is a loop? How can we visualize it, working with AI?
Part one is you fetch your own stuff as much as possible. You can see in the thing I gave before, research X. I didn’t go do the research and give it to it. I said go figure this out.
It does some amount of work. Then there’s usually a gate if you can have one, which is how do you validate that that work was correct? This is almost always the most important part. As much as possible, being able to make this deterministic is critical.
There is this idea of LLM as judge, where it can decide, and sometimes that’s necessary. But we’re seeing with this idea of mirror coding that if you have a lot of really solid tests around a piece of code, AI can replicate that code pretty agentically. The learning from that isn’t let’s replicate all this code that’s already been written. It’s that AI is really good with clear boundaries and clear definitions of done and success that it can work through. So that’s part three.
And then if you look at how it shapes, it writes out that thing and then allows it to learn and iterate and improve upon itself, hence making it a loop. If you miss the learning piece, then you’re mostly just running what I would call a skill, which isn’t a bad thing. A skill goes to a loop when you feed back the learning so it continues to improve.
Aakash: Is there a step in here, before it creates the next version I guess, between create the artifact and fetch its own inputs, where it’s pushing to GitHub or something?
Tyler: Yeah, exactly. Generally what I would do is if it passes the gate then you’d push a PR to GitHub for a human to review. If you’re just building your own skill by yourself, that would be you. You’d look at the decisions it made, how it wants to update things, and then merge that into the canonical main branch such that you’re always using that going forward. But to your point before, have a way back if something needs to go back because you don’t like it anymore.
Aakash: Are you often needing to revert back? How often does that happen?
Tyler: I tend to with AI not revert back as much anymore, because I can often do what I think of as revert forward a little easier. If I don’t like something, I almost tweak it forward as opposed to going back. Often it’s less that it’s unanimously better in the past. It might be more that there’s this thing I used to like that I’m not quite getting enough of, that I want to tweak the new version with. So I just fix forward.
What to do when the gate is a customer (14:39)
Aakash: Conceptually I get it. How does this work in practice?
Tyler: If we go next here, the AI will walk us through some of the things we do. What we do at JobNimbus to convert this into practice is we try to think about how we develop products, and I think of products pretty holistically. That could be a product for our customer, or if we’re doing something internally like using AI to develop our products or write code. How do we think of it as a loop?
If we go back to this loop of fetching its input, doing work, the gate, writing an artifact, this all works pretty efficiently if it’s code. But what do you do when the gate is a human or a customer that doesn’t work in your company?
A lot of what companies are finding out is you might have a billion ideas. You can generate ideas faster than ever, prototype them, even build them in some cases. But is that something valuable and viable for the business? If you go to Marty Cagan’s four risks, that’s still really hard to figure out. A lot of times it’s because you’re missing a clear gate. If we go back to this picture, there’s not a clearly defined gate you can quickly iterate on, because it’s your customer. You can’t just constantly lock them in a room and say, hey, do you like this, do you not like this. And even if you could, you have a bunch of customers.
So how do you manage that is a challenge. One of the ways we’re trying to adapt is, one, being way more customer centric than we’ve ever been before. Using the savings AI gives us on other things and spending it with the customer. And two, trying to mine anything they’ve already given us with the most AI or efficiency and tech we can.
We have a bunch of recordings and calls and interviews that might have been a whole team having to work through. Now with Claude Code you can point it at all those transcripts, pull stuff out, and more efficiently build that gate to at least filter out some of the bad ideas.
That’s sort of what my AI is working on right now. It’s using our product skill, and the first step is to build a bunch of prototypes. It’s going through and building some random prototypes around payment follow-up. Not really anything we’re particularly building right now, but as an example, it’s going to build me three.
I wanted to showcase that if you think about that loop doing something at the first step, while I’m not even doing anything other than talking to you, my AI is doing a lot of work to build ideas. The hard part of the loop is going to be validating that it passed the gate.
Inside the product skill (17:10)
Aakash: Where did this prototype come from? I missed that part.
Tyler: The AI is still working on it. It’s going to build three of them, variant C and B. When I said keep going, it started working on it. So this was built while we were talking.
Aakash: What did we do to kick this off? Why did it know to start a prototype? Can we look inside that product skill?
Tyler: That’s not in here because I demoed it, but I can go find it. In fact this is a good example of Herder. If I wanted to spin up my own pane here, I just say new and then I can go into our product area. Wait, not that one. Really the best one to show would be Forge.
It might be a little hard to walk through the exact skills, but if we look at all the things in here, you’ll see this .claude folder has a bunch of skills and commands that let us do a lot of this stuff.
We essentially built our own prototyping system. And by we, I mean the UX team here is awesome at being at the cutting edge. We’ve got a bunch of skills that let us do really cool things like create an iOS prototype, an Android one, port components, pull requests, run content. There’s more outside of this, this is just an example.
What these skills do is take a lot of our beliefs about how to build product and design and convert them into something a little bit more deterministic. It’s not code, but we do try to build those into canonical shapes. Then we can say to the AI, use this skill to generate 10 prototypes to solve this problem for our customers. And then we can start iterating on the feedback loop.
Aakash: This is an important point. You’re having your domain owners create skills for their specific area. The UX team is creating the prototyping skills, and I guess the product team creates the product skills.
Tyler: Exactly. And you can see it kicked out two more. We try to get it to spin out variations of an idea. These are pretty basic for speed, but you can see payment follow-up, then there’s a crew schedule one, there’s a different crew board idea, here’s another one about what you might need to know today.
The idea with AI is how do we actually test a lot of ideas very quickly, because AI is really good at that, without building.
Aakash: Yeah.
Tyler: As opposed to building the thing we think is right and then just doing the normal product cycle. If you’re mostly using AI to build in production, that’s valuable. But can you use AI to build prototypes way faster? Instantly, anyone can build a prototype and go talk to a customer about it.
From a hundred ideas down to one (19:47)
Aakash: Now that I have these prototypes, what do I do next?
Tyler: For us the next step would be, and unfortunately we don’t have this skillified quite yet in a great way, but we’ve injected all of our customer research calls and transcripts into our data warehouse. So you’ll have an opportunity to synthetically have a customer inspect the prototype through AI.
It will try to think like our customer. For anyone that might be a customer, that’s not the highest bar, but it is a good bar to catch some low-hanging failures that we missed.
The reason we like that is it lets us very quickly move through this loop. You need a gate. So we can quickly go from maybe a hundred ideas down to the five best ones. Then we start getting with our customers, setting up calls, talking to them, getting into the office with them or out on the field. So we go from a bunch of ideas to some ideas to a few ideas to hopefully the one we then finally go build and do A/B testing with in production. The idea is you lower the amount of effort until you get more and more validity of your idea from the customer.
Who gets to talk to customers (20:52)
Aakash: Who would be running these customer outreach conversations? PMs, engineers, designers, anybody? Because sometimes the UX research function in particular has a very strong point of view about who can run those and how they should be run in order to get accurate insights.
Tyler: One of the things we’re working on, and this is a newer muscle for us because traditionally it has been more of a UX and product run thing like most companies, is really trying to maintain the standards and do things right. That’s really important. But we’re trying to build the muscle of everybody doing this weekly if possible. In fact we try to aim for twice a week. We’re not there. We’re still working towards that.
If you think about where the new bottlenecks come with AI, this is going to be one of them. Getting in front of customers and talking to them. If you try to gate that too much, it’s not going to scale with the speed of AI.
So we’re trying to figure out how to really invest in our people such that anyone can start to talk to the customers and know some of the obvious traps, like leading the witness or biasing them too much. The only thing worse than no data is really bad data. So we don’t want that. But we really believe it’s possible with AI, skills, training, and leaning into the people side that we can get anyone to talk to customers and validate ideas. That is a new muscle. Some people have never done this and it might be a little uncomfortable, but we think that’s okay.
Onboarding as a codified system (22:20)
Aakash: Makes sense. So what has it done next?
Tyler: You can see as we go through this, if you don’t use Claude as much, it’s doing what they call hooks. It’s trying to make sure it does everything I’ve asked it to do.
Now it’s showing a representation of the onboarding we do here. We talked about how we really onboard with AI and want people to have a good experience. What we found is a lot of having a good experience is having a clear roadmap to success.
Stage one is setting up all of your stuff. Stage two is understanding the customer and market for where your team is. The different strategy and principles, we’ve got that all codified. The product process, the loops that exist or the skills, and actually shipping something.
These are just some of the high-level buckets, but we can basically build a simple doc that explains the person, their role, what team they’re on, who their mentors would be, who they need to connect to. We feed that to AI. In fact, AI really interviews us for that, because we’ve codified it. Then it generates a whole onboarding flow for them that they can work through.
We use Linear for our project management, so it creates them a Linear board. They come in on day one, and we still talk to them, don’t worry, there’s a lot of human elements here. But for when they’re on their own and working through what to do, they don’t have to remember from the conversations. They have clear direction. Hey, you should meet with these people. This is your mentor. This is where you start reading about the product strategy and our process.
We found that onboarding is way more efficient and also way more enjoyable, because you’re not spending so much time just trying to get up to speed.
Aakash: How do you build an onboarding like this? Basically it’s an HTML walk-me-through of getting onboarded on my job.
Tyler: What we found is AI is really good at systematizing existing process. So if your onboarding is already wild and all over the place, the first step is to write down what you want the process to be. Then you can start thinking about where AI can help.
For us the key pieces were things like setting up your laptop. Okay, cool, we can totally codify that into the onboarding. Setting up meetings with people. The AI now will just use my calendar and set up the meetings for you. That’s automated. I just tell it who you need to meet with in the first week and it finds time and sets it up.
Another one was customer research in the product, really understanding the product. So we built a product tutor. We put it into our repo. When you come in, we say, hey, go here in the terminal, run this skill, and it will walk you through the product. If you have questions, you can get answers right there.
It’s going to be different for everybody depending on where you find value. But if you can write down your system, then it usually becomes pretty clear. Okay, AI could help with this. It can’t do talking to customers. We don’t want that just happening in docs. We want you out in the field talking to customers. So we’ll have that being set up, but not done by AI. Thinking through some of the pros and cons is something we want humans doing.
But there’s a lot we found, especially in onboarding, that’s just process. It’s just getting onboarded. It’s being up to speed on everything that’s already been done.
Docs for AI versus docs for humans (25:28)
Aakash: My worry with long documents like these is that AI will create this fancy thing and ultimately you just spend more time editing it to get it to be accurate than you would have if you’d just done a simpler version yourself. So how do you build this in the right compounding way?
Tyler: One of the things we’re still working through honestly is managing that. Everyone got this crazy writer that can write anything for them in as much detail as they want.
A lot of companies historically valued that in some way, even if it wasn’t explicit. If you showed up to a meeting with a big document, it implied you did a lot of research. As humans we seem to have associated value with length.
I can’t even tell you the number of documents shot over to me that are 10 to 20 pages of AI, and I’m just like, yeah, that’s not happening. That’s going straight to my AI. Which kind of defeats the point.
So what do you do? What we’re thinking about is that something should be AI written for AI. Part of the onboarding we’ve done is not for you to read. It’s to give context to the AI to walk you through things and answer your questions. If you want to know why did we make this strategic decision, and we give all that context to the AI, that’s a lot. If we give it as much as we can, we would never expect a human to read all of that. But it provides context for the AI to answer. So that’s essentially docs for AI.
What we found now, and this is more of a newer thing, is we’re leaning towards more visual, more succinct docs for humans. AI can still help you, but you’ve got to spend a lot of time as a human really cutting it down. Maybe adding more visual components that people wouldn’t read but actually understand, like a graph of data or even a picture, that would have been really hard because most of us aren’t designers.
So now, if someone’s sending me something they expect me to read as a human and not have an AI ingest for me, I think of one to three pages as the max. I prefer it comes across as HTML and somewhat visual. If they really want a ton of details, I think it’s okay to think about AI as an interpreter for that.
Then there’s a rare case where you actually want the humans to go really deep into a 10 or 20 page document. I think that still exists. But if the person used AI and not a lot of their own thinking to generate that document, I don’t think it’s reasonable that other people will use their own energy to consume the document.
Aakash: That’s got to be the most frustrating thing, that sometimes it feels like people are spending roughly as long as it took me to read this as to write it. Nobody wants that.
Tyler: Nobody does. A lot of times I think you’re almost kicking the effort over to another person. You just keep bouncing the level of effort around, where it’s like, oh, I need you to do this thing. Okay, I’ll have AI do it and then I’ll send it back to you. And you’re like, okay, I could have done that. So what was the value add?
How to write a skill from scratch (28:33)
Aakash: Exactly. So how do we create a non-slop loop? Maybe you can help us program a loop from scratch.
Tyler: Non-slop is interesting, because you hear the word slop thrown around a lot. I would argue that some AI slop is actually better than human slop I’ve seen before AI.
One thing I like to remind people is that before AI, we did not live in a perfect utopia of only good documents and good code running around the universe. A lot of stuff was bad. People don’t like to remember, but we used to copy and paste code from Stack Overflow. It was not that different than AI. Just less efficient.
For me, when I think about writing a good skill, we could even go through this. Let me go back here. I’ll just make a temp directory.
If you’re going to make a skill, honestly one of the first things you can do, and I’ll just type here, but in a best case you’re going in here, and we’ll just do SKILL.md, which is very basic, not including any code. I would write the first pass by hand.
There’s actually a lot, well, not a lot, but I’ve read some papers that suggest that skills authored by humans are often better than the ones authored by AI. I think the reason for that is you know more and can express more of what you want this thing to do.
If you offload that to AI, you might tell yourself, it’s kind of like, you know, ebikes are popular where I live. People are like, yeah, I got an ebike, but don’t worry, I still want to get exercise and we’ll pedal. Once you get on the ebike, you’re not pedaling. That’s just the truth for most people.
With AI, once you get on the AI loop, it can be really hard to say, now I’m going to inject my thinking. Because the loop’s moving so fast. It’s like you’re on the ebike and it’s moving and it’s fun and you’re getting the dopamine. It can be a little hard to get off that treadmill.
So how do you start? I’d start like a human. And just be like, help me make good decisions. I specifically struggle, and I’m just going to make stuff up, with decision fatigue, and would love someone to use more, what’s a good word for this, contrarian thinking, but also help push me to make a decision. And you can tell I suck at typing now since AI. When enough thinking’s been done. I also like to know the latest research and have a partner that questions, pushes me, uses the Socratic method, and calls my BS. Also be succinct.
I always add this because man, if you use Opus 5, the thing likes to talk.
Aakash: The average length of responses between Sonnet and Opus 4.6, they’re in one realm, and then Opus 5 is just so wordy.
Tyler: It’s so wordy. For me it definitely regressed. It sounds more like AI than any AI I’ve used. It’s insane.
Running the skill and closing the loop (35:56)
Tyler: We can go a lot further on this and there’s a lot more depth to skills. But if I just do that and write quit, then the way Claude looks for skills is you can make a .claude folder, and we can move that skill to .claude and we’ll just make it decisions.md. Pretty sure it’ll find that. So we’ve got there. You don’t see it because it’s a hidden folder.
Then if we go open Claude. Okay. Help me decide whether to invest in robots for roofing, just making stuff up, using our decide skill.
If this worked correctly, you can see that it pulled the skill aside. So we’re starting the loop off with a skill, which is essentially something to help us start interacting with an AI in a way that we’ve defined the boundaries.
You can see it’s starting with phase zero, which I didn’t even say. It’s inferring that I need to know what invest means here. It’s using some of that Socratic stuff I said. It’s asking me clarifying questions. In a perfect world we’d actually just be talking to it, so if you use voice, that’s really efficient.
But we could talk through and say, it is reversible and the blast radius, well, maybe not reversible. Now that I think about what I asked it, it’s not easily reversible. Man, don’t take spelling from me. The blast radius is large and cost of wait is small. Let’s just throw that in there.
I picked this because it’s not something we’re thinking about right now, so it’s completely out of nowhere. And it’s a big question. So it’s saying one-way door, high blast radius, urgency. You’ll start to see it’s using tech-type words because it’s trained on all this corpus of the world. Two-way doors, it’s thinking about the blast radius, the urgency. That combination is the textbook case for not deciding yet.
I tried to purposely give it something to push it in that way so this doesn’t go on forever. So it’s saying, okay, you probably don’t actually need to decide now. Who owns this call? I can say the CEO.
And then after this, what you’d do to close the loop in the most simple way possible, and you can build all this into systems and go to level 10, is you can say: based on this conversation, how should we improve our skill to be better and more effective next time?
This is where the loop’s coming in more manually, where we’re saying, hey, we just had a conversation, you’ve seen the conversation. I don’t know what it’s going to recommend because this is a very short conversation to show the point, but it’ll come back with something. We’ll riff with it on that and say okay, yeah, let’s do this, let’s not do that. And if we go back and look at the skill, it’ll surely be a lot better and a lot more involved in the ways we failed before.
That’s really where the loop happens. And this can be hard to remember. So what I’ve seen a lot of people do is build in hooks, which is essentially, before AI closes out the session or when it’s closing out the session, it will force a kind of improvement loop. That’s something you can build in Claude. It’s called Claude hooks.
So it’s giving me some changes, and now it’s saying do you want me to apply those changes. This is where I would say, again, this is the time to get off the AI treadmill. Use your brain. Think about what you actually want to put into it. Because AI will always have suggestions on how to improve things. It’s never not going to say that. And if you always accept what it says, it will become too much. The skill gets too opinionated, too much context, too many tokens, and it dilutes the whole value.
Aakash: It tries to create mega skills that have 10 use cases in one, which is completely not what, if you look at the best practices even from Anthropic themselves, is the way to do it. The AI just really seems to like that. It wants to solve all the problems with a given skill.
Turning a skill into a loop (39:47)
Aakash: So how do we turn this into a loop? I got the basics. We had the human gate, we gave it feedback, and it improved itself. But do we want to set it off to go autonomously on its own now and start making decisions around roofing robots? What happens from here?
Tyler: You can set this up as a loop. Ask me what you need, including a learning hook before closing out sessions that learns from the log.
The one thing Claude actually supports is looping. If you do /loop, you’ll see it run a prompt or slash command on a recurring interval. I personally think loops are more than just a cron job, but one of the things about a loop is you want it to run on a schedule.
So you can tell Claude, hey, do this for me, or help me know how to do it. That’s one of the best pieces of advice I give people. If you don’t know how to do something, ask AI. That’s not always the end point, but it can often get you further along and help you at least know the questions to ask.
You said, hey, can we make this a loop? I said yes. And I told AI to do it, and I know it knows how to do this. So it’ll go set it up. It can set up the hook. It can set up the loop. It makes the skill better. And then in a few minutes we’ve gone from a one-off prompt skill to a system that loops around one specific thing here, which is making decisions around robots on roofs.
But you could imagine doing that for a lot of things. I know people that use it for preparing for the day. Every day it goes off, does its thing, takes any feedback, updates the loop, and goes back. So you come in with a fresh set of, here’s what I need to get done today.
How the prototype variants get chosen (41:22)
Aakash: These prototypes are pretty cool. How did it decide on these particular prototypes?
Tyler: If you look at this demo skill I’ve made, and this is where I’m cheating a little bit, the skill has step by steps that it’s running through. So when I told it to go next, you’ll see that it’s going through here. It’s reading the skill. It’s then got some prompts, which say to create variants for some specific prompts.
These are really basic prompts. If we go back to the actual things it generated, what I said is: create a crew schedule prototype for a contractor in a product like JobNimbus. Really basic prompt, and you see it working through that.
So the demos or the prototypes were not already built. It’s building them live. Every time I run this, it’s a little different. But the skill itself had the boundaries of this type of prototype and how many to make. You can see it’s variant A minimal, variant B full-featured, and C creative.
That actually does reflect what we do in our internal system, where we try to have at least three variants of different types. One that’s a little more full, one that’s more basic, and one that’s a little more creative. So it’s reflecting in more of a demo state, but this is exactly how we do it on our side. We just spend a little more time refining the context into the prototype, where here I just gave it a few sentences on what to build.
The loops and hooks every PM should build (42:47)
Aakash: I have a morning planning loop. What other loops and hooks should PMs be building? What’s the ideal set of them?
Tyler: One that I’ve really valued for myself is a customer outreach loop. Every day or week, or whatever the cadence is that makes sense for you, it identifies customers from your data, whether that’s from Pendo or Amplitude or a database, that you should be reaching out to. Maybe they’ve had a change in product usage, maybe they’re using a product you’re working on. You can even just have it email them, if your company is okay with that and you feel like you’ve set up the right guardrails.
It’s a really good way to keep close to the customer without one of the biggest challenges, which was just the time to go find who to reach out to and send them a message. One of the best loops I think you can do as a PM just to stay close to the customer.
Another one I see come up a lot is a project management loop. Pull tickets from Linear, Jira, whatever you’re using. See where they’re at, when they were last updated, what risks there are, who doesn’t have work in progress or needs things, and synthesize that down so you’re never having to go do all that manually.
Aakash: So I’ve got two loops. Those both sound brilliant. What other ones do I need? And also, what hooks do I need?
Tyler: Hooks that I find useful. One I already talked a little bit about, which is a post-session hook. You can set up a hook when Claude closes a session to do specific things. That’s really valuable because it’s storing all the logs on your machine, so it can inspect the logs and the things that have gone through. It’s essentially building out your own eval harness.
One of the things you can do is build a, hey, this thing finished, run it against where it ran into issues. Where did we have back and forth? What could we improve? It can just message you with that when it closes and say, hey, these are some issues. If you want to go even further, you can automate that process. Again, there are pros and cons, because like we said, Claude might over-automate and try to jam every possible thing into your improvements.
Other useful hooks are startup hooks. When it starts up, you might want to know things like what repo you’re in, where it’s at on Git, which branch, is it the main branch.
Hooks add some determinism into the looping of Claude, because Claude itself is a loop. It’s essentially asking you for something, taking it, iterating, and then coming back to you. At each phase of that loop they let you inject code through hooks. So it will always happen. Whereas in a prompt or a skill, you say never share a password, and Claude can just be like, I didn’t read that today. We’ve all experienced this, where Claude’s like, yeah, you’re totally right, you said that and I forgot. That doesn’t do a lot of good if all your data got blown out of the water.
Aakash: So you create a bunch of safety-related hooks. Just to review the key loops, I don’t know if I remember them all off the top of my head, but there’s one around work in progress and things happening in Linear. There’s one around customer outreach. There’s one around synthesizing support and what feedback you’re getting. What other ones should I be thinking about after that?
Skills that push your thinking (46:24)
Tyler: Some of the big ones I’m trying to think of on the product team that people use. This one’s less of a loop and more of a straight skill, because it’s not necessarily looping as much. We’ve tried to start building skills around thinking, so that it’s less about doing and more about pushing the thinking of the people on the team.
If you’re doing more of the product work, it’s pushing your thinking on things like, did you talk to customers? Did you look at the benchmarks in the industry for these things? Where does the data come from? Did you think about this risk or feasibility?
Because we’re almost trying to force you off this AI treadmill at certain points, as opposed to just saying, hey Claude, the CEO told me to go build this feature, can you go do all the research and tell me what to build and why to build it and blah blah blah. That’s not actually a terrible place to maybe start. But I find that because it’s so addicting, you actually just finish there. You get all this stuff back and you’re like, I can’t possibly go and evaluate this, so I might as well just send it up the chain and see what happens. That’s not doing product work. That’s outsourcing your thinking.
Aakash: That’s career reputation suicide, I feel like, for a PM at this point. Do not ship anything that AI just developed and start to pass that up the chain. You’re just offloading the work to people above you who are even busier than you.
Tyler: And to be clear, I don’t see that happening on our team. I think we’ve had some, not all the way up the chain, but there have been things that maybe got higher than you’d have wanted to see.
I think it’s not because people don’t want to do quality work. They’re feeling like they have this tool that helps them move faster, so they want to keep moving faster because now everyone’s using it. People also aren’t unaware. If you went back a year and you were doing this, people maybe were like, holy crap, this guy’s insane or this girl’s insane. Now people are like, that’s just AI. I know this is not thoughtful.
So think about it. Use AI to help you do research, to push you to think, have it even give you its ideas. But if you end up in a meeting and your answer is Claude said that, or told me that, or did this thing outside of just normal data gathering like it’s a code machine or a searching machine for you, that’s bad.
I hear that from time to time. People are like, yeah, Claude said it would take this long to build this feature. And I’m like, whoa, Claude does not know how long it will take. That was more like six months ago that I would hear that, when Claude was newer on the scene, especially for PMs.
Aakash: You’d ask Claude how long it’ll take six months ago, and it’s like a week, and you say go build it and it’d be built. It was crazy. It would overestimate everything.
Quality loops for engineers (49:03)
Aakash: I think I’ve got a good sense of what loops are, what hooks are, when I need to use them. Talk to me a little bit about going from product to engineering. What engineering loops should be out there? What engineering hooks should be out there? And then PMs interfacing with engineers. When does the PM work and PM loops end and the engineering loops begin?
Tyler: Some of the most important loops for engineers right now are quality loops. The thing people don’t talk enough about, in my opinion, is if you ship let’s say twice as fast with AI and your quality rate maintains the same rate, let’s say you have a 1% bug defect ratio or something, your customer experiences twice as many bugs.
If your quality doesn’t improve, the customer doesn’t experience your defect rate. They experience the number of defects that you push out there.
We’re seeing this. You look at big companies, their uptime is the worst it’s been in a long time. My opinion is it’s not that their quality got worse actually, it’s that they’re shipping more code and so more things go wrong at the same rate.
So we’ve really started to invest in quality improvements through loops for engineering. Our standards checks, automated end-to-end type testing. There’s a billion things you can do there. But if you don’t have AI working on the quality side, you’re going to move faster and your customers will feel like you’ve gotten worse at building things even if you didn’t, because they will experience the velocity of more bugs.
So that’s where I would start. If you’re looping, loop on quality for engineers.
Should PMs be pushing code (50:36)
Aakash: Should PMs be pushing PRs? Where does that go? Should PMs be working on engineering tasks at all?
Tyler: I think it’s definitely appropriate for PMs to work on engineering tasks where it makes sense. Let me give an example. I don’t know if I’d put a PM working on our back-end billing system and making upgrades to it that could impact people’s money flow. There are a lot of important decisions to be made there that are more architectural.
I think it’s fairly reasonable to push some front-end changes where we have good decoupling from the back end, good APIs, good CI/CD, good quality testing. The more you trust your system to do a lot of that checking, the more I think it’s okay for anybody, UX, PM, whoever, to push.
Like I mentioned before, you are at the mercy of your systems. Great systems do really well with AI. If you’re a company that had really bad systems and relied a lot on humans and slowness to protect you, you don’t want PMs coding in there, because you barely want your engineers coding in there. But with great systems, I think there’s no reason PMs can’t jump in and help.
The one thing I’d be careful of is make sure it’s agreed upon in the team, because it can create some shadow work for engineers. You want code reviewed. That’s the same for any engineer. I’ve seen PMs be like, yeah, I just pushed this, I was working on the side, can someone take a look? And now you’ve generated maybe hours of work for someone that wasn’t planned. You might get frustrated because you’re like, hey, I’m trying to ship stuff. And they’re frustrated because it wasn’t agreed upon that that would be part of their workload.
As an engineering community, this is one of the things we’re struggling with. How do we manage the code onslaught? Because you could have AI generate basically infinite code. So how do you do that? Systems is one thing people are thinking about. But the dirty truth is most companies’ engineering systems weren’t perfect before, so they’re not perfect now.
I tend to push quality, like I said, on the loops, because I almost think the quality advantage of AI can beat the velocity advantage, because it lets you ship safer.
Engineers doing vibe PMing (52:51)
Aakash: That’s one part of it, which is PMs doing engineering work. It sounds like PMs shouldn’t be trying to become engineers, but if they’re doing some front-end changes where they’ve put in probably more time on it than the reviewer would need to, then it might make sense. What about the other side, which is engineers doing vibe PMing?
Tyler: The failure points of vibe PMing are interesting, because failure points of vibe engineering can be potentially more obvious. You shipped a bug, you created a problem.
So what’s the failure mode of vibe PMing? It’s essentially shipping something that people don’t want. Or spending time going down a road that’s not viable for the business, or not feasible, or not valuable.
If you think about the four risk factors I often come back to, feasibility, engineers already know that, we’re good. Usability is generally handled more by design, so we can think about that in a minute. So vibe PMing is more thinking about, is it viable for the business, can they make money, is it the margin you need, and is it something people actually want and would pay for, the value side.
So how do you vibe PM? In my opinion, engineers are more empowered than ever to have AI help them mine data. To go out into the industry to see what competitors are doing, to look at calls, to even go find customers they could talk to. You can very easily do all that.
The risk is that they do it poorly and you go down a path that doesn’t make sense. One of the ways you can mitigate that is speed. If you have a bunch of ideas but you’re very quick to throw them away, then it’s kind of okay.
So if you’re an engineer thinking about doing more product stuff, one of the advantages you have, and where I’d suggest starting, is you can generally build pretty fast, especially with AI. Get a space where you feel comfortable that that’s a two-way door, to use the AI decision framing, and go try to do some of the PMing thing. Talk to the business. How do we make money? Talk to the customer. What do they need? Don’t overthink it. Get something out there behind a feature flag and get it in front of a customer and talk to them. That is very low risk.
It’s the version of shipping a button color change or something for a PM in code. I think there is a vibe PMing. Where I don’t want engineers vibe PMing is, hey, we need to make a strategic decision on how to invest money that could impact three years from now where this company is. That’s not vibing. That’s, let’s get deep into this thing and use the skill set.
In the same way I think of engineering, there’s this gradient. Hey, we really need best-in-class trained product thinking. But not all problems are that way, just as not all problems are really hard engineering problems.
So when I’m thinking about where I want engineers to lean in, it’s like, hey, we wanted to make this change, we heard customers complain about this thing, it’s on the roadmap for us, it’s not a huge thing, but we need to figure some of the things out. Can you do that?
The reason that’s so powerful is if you as a person can go end to end, from here’s a problem to here’s a solution that’s making the business money or the customer happy, your leverage has gone up exponentially in the company, and the speed to value went up exponentially. All those human bottlenecks internally go away, all those different decision points and getting people together.
That was a de-risking thing. We did the triad to de-risk things because things took a while to build. If you can build things instantly, you don’t need as much de-risking. You still need some, because your customers aren’t okay with constant change, and giving you feedback is not their main job. But the bar I do think has changed, because the time to build has gone down. Which is why waterfall existed, right, in manufacturing. The time to create something physical takes a long time, so you spend a lot of time de-risking it up front.
Why you never skip divergent prototypes (56:47)
Aakash: We placed a lot of emphasis on this episode too on de-risking in terms of creating divergent prototypes and putting those in front of synthetic users, then putting the best ones in front of users. How important is that? For what features do you need to do that, and which do you skip it for?
Tyler: Never skip it, in my opinion. The corollary to engineering is we used to talk about what your test coverage is, and now in my opinion it should just be 100%. It’s basically free. To tell the AI on the unit test side, that’s not all tests, but on unit tests, to tell AI to write 100% coverage is basically free. Why not do it? Before we didn’t because it cost time and energy, and there were good reasons.
If you’re doing product work, why would you not take 10 minutes and have AI spin up a few different ideas for you to consider, push your thinking a bit, do some synthetic analysis? Don’t just take what it says, but see if it’s got some points where you’re like, oh yeah, I didn’t think about that, our customers have said that. And that’s a thing I would have missed. That’s just insane to me.
We know from research, I think out of Microsoft, that a third of the product things we release won’t deliver value. No, a third will. So two-thirds of the things either are flat or don’t do anything for the business. So why would you not take some time to basically get more at-bats for free?
That’s where the synthetic stuff can be useful. That’s also where having really good relationships with your customers is so useful. Just get some more reps. Because AI makes it free. It wasn’t free before. You had to have someone design this stuff. If you’re newer to your career, maybe you haven’t experienced that, but it definitely used to take a lot more time.
What design loops look like (58:32)
Aakash: We touched on the PM and the engineering loop side of it. The other portion of the triad, of course, is design. What loops are they building? How do these all come together to vibe product develop?
Tyler: The biggest loops you’ll see on UX is prototyping. They were doing a lot of that anyway through Figma. Figma was a darling in tech, and I think that AI has really hurt them, because Claude Design came out. A lot of people I know were vibe coding internal tools, design tools, to work with their systems.
So I see design really leaning in heavily on how do we iterate faster in a way that matches how we think about design and our components. You can totally build that into loops.
I think of, what is vibe design, the one we haven’t talked about. It’s really a PM or an engineer spinning up something that’s designed in a way that wouldn’t make your design team immediately throw up. And the goal again is there’s a gradient. Some things need a lot of design and to be really thoughtful. Some things don’t.
We used to always go through design for that because they had the skills and the tools. Now with AI, if you have a good design system, it’ll use our component library, use our system. So if you want to make some changes to the front end, I think you should totally be able to do that. Having your design system know those boundaries is helpful, so when people use it you’re like, yeah, that’s something we feel comfortable with non-design experts riffing on. But you never learn this if you don’t try.
When I think about the triad, it’s really more coming together into what people might be calling a builder. And I think that coming together is going to take longer than we think. It’s not like, some people want to act like today everyone’s a builder. It takes time. It takes time to learn new skills, to build systems, to do all that stuff.
But there are people today that have found a way to merge design, product, and engineering into one person themselves with AI that are creating tremendous value. Whether it’s for a company they’re in or their own business. I bet you probably, running your own basically product catalog of podcasts and newsletters, you probably feel like you’ve merged a lot of skills that you didn’t have before.
That I think is an exciting future. And we’ve got to get there. We’ve got to train people. We’ve got to build the tools. We’ve got to invest there. It just doesn’t come for free.
But I really think designers are in a very powerful place, where they more than anyone often understand the customer, which is the hardest thing to replicate with AI. AI can code. AI can do the research. But it can’t really replicate the customer. So if you have that knowledge and you’re willing to think a bit more about the business and how they make money, and a bit more about how to build things, I think you can totally go end to end as a designer. And even if you don’t do that, your prototypes will become so much better and you can get them moving so much faster because of all these AI tools.
I’m a huge fan of this idea of design, product, engineering coming together. People will still have expertise, for sure, and that’s important. But it’s just fun to work across the whole stack on certain problems that make that a reality.
Hiring full-stack product builders (1:01:44)
Aakash: Have you hired any of these full-stack product builders, or are you still hiring specialists?
Tyler: We hire a lot of them actually now. We hosted a hackathon. I think it’s the largest hackathon in Utah when we did it. It probably still is, because it wasn’t that long ago. We got hundreds of people to show up.
One of the people that won it is now an SVP at our company, and he’s a builder. He built a product with a small team end to end in like two days. We asked them to build an actual product for our user base and it was crazy. And then another person that I think got third, we hired two onto a team.
What we’re actually finding is not that skills don’t matter. Engineering skills matter, design skills matter, all these things that people have learned matter. But if you have the ability to take those skills and then inject AI in the right way and push a little bit maybe outside of your traditional comfort zone, it just opens up a world of opportunity for you to create so much value that wasn’t really possible before.
So I’m super pro this builder mentality. Some people take that as, oh, he doesn’t believe skills matter. I think they matter more than ever. But because they matter, and with AI and systems, you can actually upskill yourself more than ever.
Should everyone become a product builder (1:03:00)
Aakash: So should every PM, designer, and engineer be aspiring to upskill themselves into a product builder?
Tyler: In my opinion, yes, unless you find yourself in a very specialized role. For example, on engineering, if you’re a security engineer, or you’re working at database level research at Google on how to build the next high scale, high throughput backend or database or whatever it might be, those types of specialized roles really need special people with special skills that are going to always be important.
A lot of SaaS companies don’t have those. They tend to generalize better. And if you can become a product builder, they will love you, because they want to solve problems that people will pay for and love using. Often that’s not an incredibly complex engineering problem. It’s not often an incredibly complex design problem or an incredibly complex product problem. It’s the merging of all of those things in the right way that’s the hard part.
If you looked at most companies’ codebases, no one would be like, wow, this is the Picasso of code. And most things are not designed in a way where we’re like, wow, put this on my wall because it’s so beautiful.
So it’s that super nexus of all those things. That took three people before, and maybe it takes three people forever, I don’t know. But one of my bets is it doesn’t, for all things going forward. I can’t predict the future, so I’m not putting money on it, but that’s what I would say.
How leaders carve out time to build the skill (1:04:23)
Aakash: What’s amazing is how quickly you ramped up on all of this, even at the C-suite level, when you probably have to spend 90% of your time in meetings and with stakeholders. So for the other leaders out there, how do you ramp up? How do you personally find the right time to build this skill set? Because if you’re going to start asking this of your team, you obviously have to lead by example, like you are doing.
Tyler: One of the things I benefit from is I’ve always enjoyed tinkering. I spend a lot of my free time outside of work doing stuff. So that helps. I won’t lie, spending extra time learning is always a good thing.
The thing I think we’ve done that’s actually been really good, and our CEO really pioneered this, is we’re in person Tuesday through Thursday. What we’ve done on Monday and Friday, which is remote, is we really discourage meetings. It’s the first place I’ve worked at that’s done this somewhat well.
So usually on Mondays and Fridays I don’t have any meetings. My team’s almost 100 people, so that’s not a bad place to be if you’re leading a team of any amount of size, where usually you’re just moving through meetings. Let me tell you, if you get in a room for eight hours with a lot of time and AI and a lot of things you want to learn and explore, you can do a lot of damage. It’s really cutting that time into your schedule.
Most people don’t have that luxury or that culture. What I would recommend is treat it as important to your role and to the adoption of these tools. A lot of companies are doing research on this, like McKinsey. Saying things like use AI doesn’t do nearly as much, if anything, versus this is how I’m using AI.
If that’s what you need to accomplish in your role, the best thing you can do is use it and show the team you’re using it. Then you start to learn where the failure modes are too. Because often high-level leaders can talk about things they read on X or your newsletter. They’re like, hey, they’re talking to this guy Tyler and he said everyone’s building, why aren’t we building? And there’s just so much nuance, even in what I’m saying about how we did this, the pain, what we’re still learning. You really have to experience it yourself to be able to speak to it.
Aakash: I feel like every leader now needs to just carve out some tinkering time.
Tyler: And most leaders love it. It’s so fun.
Aakash: It’s way more fun than being in meetings all day.
Where to find Tyler (1:06:44)
Aakash: This has been amazing. If people want to learn more, if they want to find you online, where do they go?
Tyler: Biggest place, Substack. I have a newsletter where I write about my tinkering. It’s called The AI Architect. You can find my name, Tyler Folkman, or The AI Architect.
I am on LinkedIn. LinkedIn I feel like has suffered from AI more than most places, so I do post there, but I’ll be honest, I don’t read it as much. If you engage with me there it takes me a little bit longer. But reach out. I love chatting with people.
One of the things I’m really passionate about is community and bringing together people and just having a good time. I really love AI, local AI, tinkering, building. So if that’s your thing, definitely reach out. And if you’re in Utah, we’re always looking for great builders. JobNimbus is a super fun place to work.
Aakash: Sounds like a place where you can actually improve your AI skill set instead of being stuck in meetings, which is another bonus. Subscribe to The AI Architect. You’ll see the link in the show notes or down below in the description. Tyler, thank you for being so generous with your learnings and time today.
Tyler: Thanks so much for having me. It was super fun.