Check out the conversation on Apple, Spotify and YouTube.
Why TinyFish lets anyone ship to production (1:32)
Aakash: TinyFish AI is a startup that has been on a hockey-stick growth path. They raised $47 million from ICONIQ. They went from 100,000 to 200,000 users in 18 days, and from 200,000 to 300,000 users in just 10 days. After I talked to Shuhao Zhang, their co-founder and chief product officer, I think a big part of the reason is the way they build. They allow anyone in the company to ship to production.
That’s because they hired a bunch of amazing former Meta engineers. Shuhao himself was a former VP of engineering, and they built the system that lets you ship features. They’ve invested in the coding agents, the code review and the CD system. Everything you can think of, from how you start a PRD all the way to how you measure the results of a feature, can be done via agents. This is the next-gen way of building products.
In today’s episode, Shuhao is going to walk you through every step of that life cycle. He’s going to pull up his real GitHub and show you how he uses Claude Code, so that you too can let anyone at your company ship features to production. Shuhao, thanks for being on the podcast. What do people really need to know about this system, and how can they build it themselves?
Shuhao: Let me show you. Aakash, today I’m going to show you the whole setup I use on a day-to-day basis, from building to shipping into production. And now I’m actually building the part that lets agents operate and maintain it, as we just talked about.
A Claude Code setup you can reach from your phone (3:07)
Shuhao: This is my Claude Code setup. I have a dev server, and this is a tmux session, so I can open it up on my mobile phone or in my browser. When I close everything, it still runs in the cloud. It’s a full EC2 server, so it can install the tools and the whole stack as needed.
You can see I have multiple projects going on here, and each of them could be a feature or a new product I’m building. This one is still live. It’s working. It’s an AI gateway product we call Tiny Router, which puts our models behind one gateway with a discount. The first part is already built, and I’m trying to land it in our sandbox environment. I can talk about the whole setup.
Aakash: Maybe you can break down some of what you just said for those of us who aren’t as technical, the “for dummies” version. Dev server: what are we looking at there? Sandbox: what does that mean? And what is this software? If you can break down those basics for us, that’d be great.
Shuhao: Sure. A dev server is essentially a remote server running in the cloud. It can be an EC2 instance in AWS, any server you host, or your own MacBook with external connectivity. The goal is that you can run your workloads without keeping your laptop running, which is a local environment.
A sandbox is a pre-production environment. The setup is very close to production, but you don’t have real user data. You have test data, so you can test your software before it goes live in production.
As for the software I’m using, I built it. Actually, Claude Code built it. It’s based on tmux, so you can stream a terminal session over the network, and it’s protected by encryption. So this is my primary way of working with Claude.
Why not Claude Code on the web? (5:29)
Aakash: Why wouldn’t you use Claude Code on the web?
Shuhao: I noticed Claude Code on the web now has more of the features I need. But I built this last year, when those features weren’t available out of the box. I definitely see a future where it’s just out of the box and I can use it. But I also like the flexibility of owning my entire instance, so I can install whatever I need. For example, if I need VPN access, I can install my VPN profile here. There are still things where Claude Code on the web isn’t flexible enough, and I’d love to have full control.
Reviewing the agent’s plans in Claude artifacts (6:10)
Shuhao: You can imagine this is the interface where I talk to the agents. But as everybody who uses Claude has seen, a lot of the UI is the agent’s own work log. The valuable communication, clarifying product requirements and giving feedback, needs to happen in a different place.
What I do is use Claude artifacts. They’re a great way of communicating. Claude can generate a plan for me to review, and I can type my response in the artifact. Let me show you one of those artifacts.
For any bigger design decision, I usually ask Claude to generate a plan, give me some recommendations and give me a status update. So this is where I review the plan. I won’t show the entire thing, but you get the idea. I can respond inline, and it picks up my response and carries on.
That’s pretty much the basic setup. You can talk to Claude, review the plans and clarify the designs. I think pretty much everybody has their own setup, and it works pretty well. What I want to talk about more is how to push your product into production, and then how to operate it in a production environment. By the way, this is something I’m actively building and experimenting with. So I’d love to hear your thoughts in the comments, and if I can update you in another episode on how things are working, I’d love to do that as well.
Aakash: Aren’t we all?
When agents write the code, review becomes the bottleneck (9:55)
Shuhao: To get into our production environment, we start with the sandbox again, and we go through the code review process. At TinyFish, we have quite a few engineers from Meta, so the code review culture is really strong. We hold every line of code very seriously. However, when agents can generate thousands of lines of code, tens of thousands of lines of code, in the blink of an eye, the human reviewer becomes the bottleneck.
So how do you solve that? We don’t want to let go of quality control. We don’t want to lose the second opinion, the third opinion. What we did is onboard CodeRabbit. I believe a lot of users love them. But how do you let CodeRabbit auto-approve your PR and still stay compliant, and still get the back and forth you usually get with your engineers?
What we do is let CodeRabbit approve the PR. But just like a human reviewer, it can also reject the PR or request action, and the agent picks that up automatically and they go back and forth. If CodeRabbit detects a high risk that requires human judgment, it loops in a human. The agent tags the right engineers to do a human review. But if CodeRabbit is happy and it’s lower risk, the PR is auto-approved and auto-merged.
Let me show you. This is quite an extensive PR, a UI change with a lot of changes and a lot of files touched. But it’s considered lower risk in general. Still, you can see quite a few responses from CodeRabbit. Here are a couple of examples. It requests changes, meaning there are things it’s not happy about, and this is where we pay attention. This is where the AI reviewer considers things not optimized or not correct. And this is my agent picking up the feedback and making a correction. Finally, the AI reviewer approves it.
You can see there are multiple rounds until both are happy with the results, and the PR is merged in the end. So this is one example of what to do when you produce a ton of pull requests. It’s a fast and safe way to get them reviewed and merged.
Aakash: So basically you have an independent tool. In this case, you’re using a third party, CodeRabbit. CodeRabbit has some context around your codebase, your coding standards and what your engineering team likes things to be like. It acts as them, and it works with the Claude agent that submitted the pull request, back and forth, until it’s happy.
The constitution behind the AI reviewer (13:04)
Shuhao: Yes. We have the concept of a constitution, which holds our conventions, our standards and how strict it should be. As you said, CodeRabbit knows the context, and it knows how far and how deep to go in the review. Because you do get into this infinite spiral where it gets too nitpicky and picks at all the problems, which isn’t necessary. You really want to catch the ones that matter.
Aakash: Yeah, you don’t want to just burn a bunch of tokens on this kind of back and forth.
Shuhao: They can really keep going if you don’t have that. That’s why we built a knowledge base, a repository of review styles and coding styles. It’s quite extensive. This is where I think modern engineering teams should invest.
Aakash: You built this, and it’s your organization’s playbook for both your coding agent and your review agent. Who’s the right person to own it? How do they execute on the project, and how much ongoing effort is required? Is it one of your most senior tech leads who’s super AI-pilled and loves this stuff? They get a project for some amount of time to work on the code reviewer agent, and then they maintain it every couple of weeks for a couple of hours? How does that process look?
Shuhao: Yeah, exactly. Our head of engineering, Pasha, personally looks into AI productivity. We have a channel where all the engineers can chime in, but one senior engineer looks into this and experiments across multiple projects, in production and in staging, to validate the approach iteratively.
Start every project from a template (14:54)
Aakash: How can people replicate this on their own? What do they need to build this up from the ground up, so they feel like they’re actually shipping production-quality features?
Shuhao: The best place to start is a template. That makes it easy for your team to replicate, and your lead engineer can work on the template. This is ours. You can see we use Terraform for every infrastructure process and for CI/CD.
What’s really important in this template is the infra folder. You have, say, a sandbox environment and a production environment, and template Terraform files that describe the requirements. You can configure your network in production. You can configure access. How do you do ECS? How do you do Kubernetes? This gives your coding agent a cohesive picture of how your cloud and your ops are set up, and it uses the same setup across different projects. Then there’s how the database should be set up, the Docker containers and your app logic, which lives in this app folder. There’s also how your tests are set up, and all the scripts.
With this template, a lot of the scaffolding is already in, especially the deployment and CI/CD part. That’s usually where the agent can be very opinionated, and you don’t want a ton of projects doing it different ways. So this is usually a good starting point to make sure your project is deployable to your environment.
Aakash: You mentioned Terraform. If people don’t know it, what is it and what does it do?
Shuhao: Terraform is a configuration language from HashiCorp. Essentially, your infrastructure is code. If you want to launch a new instance, or point a domain at a certain service, you make changes in the Terraform file instead of going to the AWS console. The benefit of that is you can have your team review this like code and double-check it. Once it’s merged, the change propagates all the way to AWS.
Aakash: Very cool.
What goes in the .claude folder (17:24)
Aakash: So I’m guessing inside this .claude folder is your CLAUDE.md, your different coding skills, the architecture of your database like you mentioned, and Docker. What all goes in it?
Shuhao: You can see it from the folder names. There are best practices for the entire organization, and there are the rules, which are the guardrails the agent must follow. This is what we want to share across the company, so everyone has the same coding agent style. It’s proven to be very effective. We’re sharing organizational knowledge and putting the best practices all in one place.
Aakash: Does everyone use the same coding agent, or is this built in an agent-agnostic way? If somebody prefers Codex or Claude, can it work in either?
Shuhao: It’s quite easy to migrate Claude skills and plugins. We do have engineers who prefer different coding agents, and we also have a tool that can transform the skills from one to the other.
From idea to PRD to engineering plan (18:35)
Aakash: You use Claude Code mainly, though. So where do we go from here? You’ve got a GitHub project. Actually, let’s start at the feature level. How are you initially prompting the feature and creating the prototype? How does all of that work?
Shuhao: For a new feature, it depends on how big it is. But I think today you also have an agreement with your agent on what kind of language you’re going to use. I prefer very simple language, and it talks back to me in simple language. Karpathy talked about the standard used in aerospace, where there’s no ambiguity and you can only use a certain vocabulary. I really like that.
My workspace is actually this. I have a text editor, and it also has a voice mode. You can see I can talk to it even while I’m driving, and it talks back. So there’s this back-and-forth brainstorm with the agent to get the context of what I want to build. I can type here, and the voice ends up here too. Then I let the agent design the top-level product design documents as an artifact. That’s where I go from there.
I also want to do enough research on the market: the competition, what’s the best product out there, what’s missing. So a significant amount of time is spent just ideating and making sure the agent knows exactly what I’m trying to build. Along the way, I encourage agents to challenge my idea, to propose alternatives and to have small-scale experiments, to fail fast and see some early flags. But the end product is a bunch of documents. You can see how many documents it produces.
Let me show you one of those. This is a scratch pad, a repository just for remembering all the designs. You can see the API design and the cloud architecture, and as I go through, there’s a whole bunch of documents. This is really the PRD, the engineering plan and the implementation plan. I won’t click through it, but each plan is sophisticated enough, and you can see the v2s. So I do go back and forth with my agent, and I know almost exactly what it’s going to do and what the design decisions are. I always ask it to list the 10, 15 decisions that need my attention, and I also encourage it to make decisions on the low-risk ones. That kicks off a fairly lengthy coding session.
Aakash: So you go from an idea you bat around with the agent, where you ask it to poke holes in your thinking, to eventually a PRD, then an engineering specification, sometimes a prototype, and then you go to code. Is that the life cycle?
Shuhao: Yes, exactly. I tend to front-load a lot of the designs and considerations and have it write the entire engineering plan and implementation plan, so it can go off on 20-plus hours or multi-day implementation and go as far as it can. Then I start another iteration of testing. I do want to play with the service myself and give some feedback. But the idea is really that I want to run multiple agents building multiple products and features in parallel. While one agent is running for hours, tens of hours, I can focus on brainstorming another feature.
Front-load the design to avoid AI fatigue (22:32)
Shuhao: With the pattern of having an agent ask you a question every half hour, what I found is that you easily get this AI fatigue. You find yourself drawn into multiple agent sessions, being the bottleneck for answering questions and approving the next action.
Aakash: That happens to me all the time. I block out two hours to do this thing, and then all of a sudden I find I spent the last two hours looping through my 12 different agents and giving them prompts. How do you make sure you stay productive in those sessions and eventually accomplish what you were there to do?
Shuhao: I really want to find a way for the agent to work continuously for much longer hours without drifting. That’s why I front-load a lot of the documentation process, from ideating to the engineering design to the implementation design, and I review those pretty extensively. So you focus on design with your agent, which is where you spend a lot of your own human time. Once you reach agreement, this particular feature is ready to go. You’re spending continuous time designing and gaining feedback, and the agent will spend continuous time building.
Why operating in production is the next frontier (23:52)
Aakash: All right, where should we go from here? What else should people know in order to build like this?
Shuhao: We’ve seen how to kick off a building session, have the code reviewed and get the PRs approved. Eventually you use Terraform, or whatever CI/CD you have, so your service, built by agents, runs in some environment, say the sandbox. Eventually it will run in production. This is where I see the next frontier: letting your agents operate.
What we just talked about is going to be fairly today’s commodity. I think everyone has their own way of doing things and their own setup. But having the agent operate in production is really something I believe will differentiate how fast companies can build their products.
Aakash: All right, let’s see it.
An observability stack your agents can read (28:05)
Shuhao: What we believe is that it’s going to be data-backed, and the agents need access to the production metrics, traces and logs. Where we started first is building a system. Let me show you this observability setup.
Every company has its own stack, and I’m not opinionated about any of them. But we decided to build our own observability stack because of the load and traffic we have, and also for control. We use a combination of open source and self-built solutions. In this diagram, you can see applications plug in really easily. It’s a couple of lines, and then your application has the entire TinyFish observability stack. You have the logs, the metrics and the traces, the three critical pieces of information you want to gather from production, and they fan out into the backend. It’s a very efficient, high-throughput system.
What’s critical is this. Usually it’s a human who consumes the observability output, whenever there’s an alarm or there are errors, or when you just want to look at the metrics and performance logs and find where to optimize. We really believe AI agents are better ones to consume this huge amount of data in production continuously. That means you watch it 24/7, and you can know where your memory has a bottleneck and why, before it even sends you an alarm and the service goes down. With any of the logs, you can watch for business logic problems, errors and opportunities to improve your system. So this is the system I really want to build well and make work at TinyFish.
Logs, metrics and traces, explained (30:14)
Aakash: Can you say more about the difference between logs, metrics and traces, and what exactly those are, for people who don’t know?
Shuhao: Logs are where you keep a more verbose version of what happened in your service. Developers can inject them. You can print out exactly what happens across, say, a user’s life cycle when they start using a feature. The service says, “We start a database connection, and this is the connection I got.” Then the user sends a particular request, and you can log the request and what your system does at each step. You can be as granular as you want. You have full control. Logs are a huge amount of data. You can think of them as the replay, if you will. If anything goes wrong, you want to see the logs.
Metrics are where you see the performance. It’s all numbers: the number of requests you can serve, the CPU load, how memory is doing and all the latencies.
Traces are how your systems connect together, how one user’s behavior goes through your multiple services. When you have a ton of microservices, how do you trace the entire activity across them? Traces are usually done automatically, through injection. All three are important, and you should look at them all together.
Aakash: Got it. And basically, what you’re showing here is a system that figures out the errors or the mistakes? What is this accomplishing for us?
Shuhao: Anything that’s a surprise or a potential improvement. If anything is wrong, you usually have an alarm system that tells you a certain run or execution didn’t complete successfully. Then you go back to the logs to see where things failed, and that can point you to the part of the code you need to fix. The metrics tell you about performance problems, how well your systems scale and how reliable they are. The traces tell you, when all the systems are connected together, where the bottlenecks are in certain microservices. If the overall latency is high, where is it? Is it the login? Is it the database? It’s a massive amount of data we’re talking about, terabytes or petabytes, and you can mine it for opportunities to improve your software and service.
The ops agent that fixes production (33:09)
Aakash: Where do we go from here?
Shuhao: Where we go from here is to have the agent look into these metrics, logs and traces continuously. These are usually behind firewalls and authentication, so you need your agent authenticated to get access. If it’s all behind the firewall, you can install the VPN on your dev server so the agent has access. Then you build a skill so it knows how your observability system works: where things are, the formats and all the mechanisms for access. Then it starts to detect problems.
Here’s an example. I’m looking at a Linear project, and it’s brand new. It’s actually running today. You have this AI ops agent detecting issues from the observability system. This is a real issue. It found that when an LLM call reaches max tokens, we return a server error. It’s actually not a server error. The server is functioning. It’s the client sending too many tokens, so it shouldn’t be that kind of error. It only detected this because it found the behavior in the logs and the traces, and then said, “Hey, this is a problem.”
What the agent does, and I’m calling it an ops agent, is open the Linear ticket first, mimicking our engineers’ behavior. Engineers would open a Linear ticket first. Then it goes ahead and tries to fix it. So this documents what it found and the potential fixes, and then it starts making the fixes. So far so good, right?
Aakash: Okay, makes sense.
Shuhao: After it makes the fix, it pushes a pull request. It pushes the code into our GitHub and asks for a review. You can see this particular fix is considered lower risk, and CodeRabbit automatically approved it. Then the agent knows, “Okay, my change is good. The tests are all okay. I’m going to merge it.” Merging here means it’s automatically picked up and deployed to the staging environment through our CI/CD.
So you see the entire thing running continuously. It watches the production traces and logs, detects errors, makes a fix, gets the AI reviewer to review it and automatically deploys it to the environment.
How the agent knows what’s wrong (35:59)
Aakash: And how does it know the North Star of what’s true, so it gives a correct answer?
Shuhao: When you look into observability, the whole mining part, quite a few of the issues are pretty obvious. You have an exception, an error log, or a super long latency that’s abnormal. What happened over the past decades is that it would either send an alarm to your on-call engineer or sit in the backlog in your error system. You use Sentry or other systems, and engineers try to crunch through the backlog. But a ton of those are never fixed, because they’re a very low percentage and hard to reproduce. The system degrades over time because of this, and users have to wait much longer for things to be fixed if they’re not the highest priority.
With the ops agent running in production, we’re talking about having multiple sessions make a fix, test it properly and deploy to production automatically when the engineers are sleeping. You can also have a gated rollout. You can roll out to 1% of the users in production and make sure there’s no regression. You can have multiple flags and A/B tests. So this is the whole new paradigm I’m looking into, and I really want to install it at TinyFish and prove it works.
Roll out every change to 1% first (37:34)
Aakash: Those things you just mentioned are pretty interesting safety mechanisms. How often should you be doing the 1% rollout? Should you always do a phased rollout, or always either a rollout or an A/B test? How do you think about shipping things that, after the engineering spec, were essentially done by agents, just Claude and CodeRabbit?
Shuhao: If you have the bandwidth and capacity, you should always do this partial rollout, always, even for a critical fix. You take an extra hour. It also depends on your production volume. You want to take an extra hour to see how this 1% of users behaves with the new change. That’s always the last defense against something catastrophic.
But it used to be very, very costly. Imagine a company with dozens of engineers, hundreds of engineers, and all the changes they make when they roll out to production. It’s not realistic to have a partial rollout for every change, because some human needs to look at the partial rollout stats and make a judgment. But now you can actually have an agent do this. So the whole thing is designed for agent automation, and arguably, with this approach, it’s safer and more efficient than human operation.
From vibe coding to agentic engineering (39:10)
Aakash: Is there anything else people need to know to replicate this on their own?
Shuhao: I think there are existing services doing this. We came from vibe coding to more of a coding agent, which removes the vibe part and gets more serious about production. Then there’s the concept of agentic engineering that we’ve started to look at more, where the agent can do not just coding but the whole thing we just talked about. I believe this is going to be what engineering focuses on.
As I said in the beginning, it no longer matters what product you build. This is the muscle every engineering organization should build: the machinery to produce, deploy and operate your product in production. You can still use as many third-party tools as you like, just like we use CodeRabbit. But I think the knowledge, the methodology, the framework and the opinions you have in the next six months will be the differentiator. Maybe in one or two years this will just be out of the box. Maybe the cloud providers will just have it. But for the time being, when everybody is still racing to the market and to their users, this is a really critical, high-ROI investment.
The biggest mistake: thinking too much (40:41)
Aakash: What are the mistakes you faced along the way, building toward this? What’s the scar tissue other people can learn from?
Shuhao: The biggest scar tissue is really thinking too much. It’s kind of a paradox, kind of self-conflicting. You want your agent to run as much as possible, but at the same time you’re thinking of all the things that could go wrong and spending a lot of time telling yourself not to begin. I think you really have to believe in this future, because it’s going to be the case. Don’t wait for others to hand you this system for your company. You really need to build it yourself. Just don’t spend too much time convincing yourself not to make a move. I think we spent enough time debating, and now we feel we could have started this a bit earlier.
Can bigger companies do this? (41:34)
Aakash: People are saying in the back of their minds right now, “Well, he’s at a startup. They’ve only raised $50 million.” So put your Meta hat back on. When you’re at Meta or a much bigger company, what are the things people say about why they can’t adopt this, and how would you address those one by one?
Shuhao: Actually, I found that companies with much bigger resources have gone really far. They’ve gone further than startups. Not all the big companies, but this is a really good fit for a company like Meta. They built their own repository infrastructure, the monorepo, things they share externally. They have an entire organization, hundreds of engineers, just working on the infrastructure piece, and this is definitely the best infrastructure to build today. So large companies with more resources actually went pretty far. They have the production volume to do these partial rollouts and get the feedback, and they have so many products and features they can experiment on.
Aakash: But many people are probably watching who aren’t at companies that have this at all. Their engineering org might be accelerated by AI, but the product org isn’t feeling it at all. The product org isn’t suddenly submitting PRDs and getting them built by agents.
Shuhao: Every technological evolution resets the market. This time, the way it resets is not only about who builds a product people love. It’s going to be about who really adopts this new way of building faster. And of course, the ones holding back are going to have a very tough time in six months, when everybody else is on a rocket and you’re still walking.
Building fast doesn’t mean shipping slop (43:24)
Aakash: Why is building fast in particular so important? In the traditional Marty Cagan school of thought, it was always that building the right thing is more important. If you build fast, you could just ship a bunch of slop, and then it becomes a product experience people don’t like.
Shuhao: First of all, building fast does not mean shipping slop, right? We’re talking about production grade. That’s the tough part. That’s where you need a lot of your product and engineering people, especially the senior ones, to think about how to ship and operate production-grade, best-in-class services and products using agents.
And why does building fast matter today? Think about why building the right thing was important in the past. It took maybe six months to get a PRD locked, with every director’s signature. I’ve gone through that time. Then you have the engineering design doc, the ERD, and that’s another three to six months, with the architects designing. Because building was so costly back then, you wanted to make sure you built the right thing. But today building is so fast that you don’t want to spend a year deciding what to build. You want to put it out as fast as possible. The best way to get feedback is not a room of PMs and engineers sitting in your own company. It’s from the user. So things have kind of reversed.
What a quality PRD looks like for agents (44:55)
Aakash: Can you show us the right level and quality of input a product engineer or product team needs to give, so an agent can go out and operate correctly? Maybe what a quality PRD looks like?
Shuhao: This is a research agent. I can quickly scroll through the definition, and you can see how sophisticated it is.
Aakash: So again, this is quite in-depth. What I worry about when AI writes these long documents is drift. How did you create the right alignment between what you wanted and what that document said?
Shuhao: You layer them. I have the PRD, and I also have the engineering plan, which you can see is actually responding to the PRD. Then you have the implementation plan. Again, this is something you want your agents to work on for hours and hours. Agents today are getting better and better at planning a large-scale project, and they’ll have their own sub-agent planning: “Okay, I’m going to build one component, and this is what I’m going to focus on.” Then a bunch of sub-agents create these components and piece them together at the end. So more granular, more detailed planning up front does help.
The reason drift happens is that agents start to make assumptions when things are not discussed, not documented. But when you have enough context, it’s no longer a problem. It really knows to read a certain part of the document, go back to work, and come back to another part of the document.
From staging to production (46:57)
Aakash: To complete the teaching you were doing for us earlier: you were going to push the initial feature to staging. What happens next? Where are the steps where a human is in the loop?
Shuhao: After you push this to staging… Staging is an environment almost identical to production. Companies usually invest in building their staging environments to simulate production, so you never mess up users’ data and experience. Pushing from staging to production is actually a lot faster and easier after you’ve done enough testing, because the environment is identical. You flip certain flags, do the gradual rollout, the partial rollouts, and that’s it.
I spend more time in staging. You dogfood it if you have a UI component in your product, or you load test it if it’s a backend service, and again, AI is very good at that. When things look okay, you consider rolling out to production. And that’s only the beginning of the journey. Then the agents start to watch the observability and make improvements again.
Where a human still has to try the product (48:16)
Aakash: What about the part where a human would typically look at the results of the feature and say, “Do we really want to keep this in the product? Do we want to do more here?” How much have you brought agents into that part of the process?
Shuhao: As a human, you definitely want to experience the product, no matter what it is. Let me show you. This is one of our products, the model router I talked about. Initially, the color scheme wasn’t this. It was green, and the TinyFish color scheme is orange-ish. So that was a miss in the initial design with the agent. This is the part where you really want to go in, play and see if the playground works when you choose different models, and see how the latency feels as a human. The latency is all captured in the traces and the metrics, but you still want a human touch. You want to try different things.
We also have a test automation agent that can open a browser and click around. Because TinyFish is a web agent, we use our own product for UI test automation. But a human has to experience it and dogfood it, and then all the findings can be collected and sent back to the coding agent again.
Test automation, explained (49:45)
Aakash: What we’ve done so far in this episode is really help people with non-technical PM backgrounds get into all these technical concepts. Can you also explain test automation, for those who don’t know, and how it all works?
Shuhao: For all the code that either engineers or AI agents create, you need a way to test it, and you don’t want to test everything manually. It’s just not scalable. So usually you have anything from unit tests to end-to-end integration tests, and then load tests.
Unit tests cover smaller features, like whether you can do one plus one correctly, and you have a ton of unit tests covering the low-level logic. End-to-end tests are more about what the experience will be as a user. They won’t capture all the problems, but they give you a good idea of whether a user will easily run into problems or whether it’s going to be okay. And load tests, of course, check whether your system can hold when you have a large volume in production.
AI agents are actually pretty good at creating all of these tests. You just need a different set of infrastructure. Unit tests are the most straightforward. They’re just another set of code. Load tests bring in a simulation of traffic. And for end-to-end tests, you need some sort of agentic solution that can open a browser and navigate.
How technical does a PM need to be? (51:13)
Aakash: Wow. Okay. So how technical do people need to be to use a system like this? There’s probably some part of it where you need a really deep engineering background to go this technical. How technical does one need to get? I imagine they can partner with an engineer who handles a lot of this.
Shuhao: If there’s one takeaway from this episode, I really want all the product builders, PMs, and people interested in becoming one in this AI era to hear this: don’t be intimidated by the amount of technologies and engineering requirements. Pushing things into production sounds very scary when you have a large user base. It’s really a reset.
But you do need a good engineering partner. Like you said, if you don’t know how to drive, you got to have a friend who can drive. It’s really about knowing the high-level risks, what good looks like and the common practices. The unknown unknowns are usually very difficult. But if you have an engineering partner, they can easily tell: “Okay, this is someone who doesn’t have a technical background. What are the areas they may miss? Let me fill that in.”
When you have the right setup, hopefully your engineers are very supportive of the system. The ideal case is really that the PM will spend a lot of time talking to the agent back and forth, define the requirements and then give feedback in the lab test. The rest will be taken care of and supposed to be taken care of by your company’s machinery.
Who owns a feature after launch (53:06)
Aakash: We started at the beginning with the idea that when you shipped a feature, you’d hold an engineer hostage. They’d have to maintain it, know it and be accountable for it. Is that now shifting to the PM? Is it the system? How do you make sure that accountability still exists somewhere?
Shuhao: It’s a spectrum. Traditionally, when you build something, you’re responsible for it forever. Even if you have a dedicated support team, when it comes to the tricky ones, it always comes back to whoever created the service. And then you hit a limit. You start realizing someone spends half of their time maintaining and supporting what they built. Eventually they may only be able to run two services, two features at most, which is what I mean by holding someone hostage. It’s not supposed to be that way. You always want to create, improve and innovate.
So agents are really going to play that role of keeping the lights on and continuing to improve. I don’t think it’s the PM’s job either. The PM is going to watch the market reaction and the top lines, and see how we can enrich and improve the products and services, not spend a lot of time on the low-level stuff.
Where the PM role ends and engineering begins (54:40)
Aakash: Just to put it in concrete terms, where does the role of the PM begin and end in this future, and where does the developer’s begin and end? If the engineers are building the system that lets PMs build a bunch of features, one could forecast a future where engineers focus more on the infrastructure, the system to build the system, and PMs focus on the inputs to the system. So the amount of engineering you’d need to build a particular feature would be less. Is that true, or have I played it out the wrong way? How would you react to that?
Shuhao: I think you’re correct. But we’re also in this amazing world where we don’t really know what the models can do in six months, so whatever we say today could be different. What I’m envisioning is that it’s still the best time, and it’s going to be an even better time, for both engineers and PMs. Think of the amount of work these two professions did in the past that wasn’t really adding much value. 90% of that will be gone.
Today you’re seeing PMs, or anyone, build something that looks amazing in a demo. But there will be an inflection point where PMs go back to focusing on the market and the users and creating the right things, and building becomes even invisible. Today you still need to prompt, do a lot and work with Claude Code. But I believe that in the near future you won’t really need to see the Claude Code session anymore. The PM is going to use all the tools to know where the market is going, and the engineers will make sure the machinery behind the scenes is very competitive in whatever landscape you’re in. So they’re both going back to what they’re really good at.
Today we’re seeing a mix. But I don’t really want to mix up building a prototype and putting it out for a little while with real building. If a PM feels great about “Okay, I can use Claude Code to put something out, I don’t need an engineer,” I think that needs to kind of change. You still need your engineers to think much bigger, and you partner together to build this machinery.
The cost of not building (57:12)
Aakash: One of the most interesting product philosophies you shared, and I’d love to hear the next layer of the onion on it, is this idea that the cost of building used to be astronomical, and now the cost of not building is astronomical.
Shuhao: There are so many reasons we used to spend a lot of time finding the right things to build. We talked about that just now. But today the cost of building is so low, and it’s going to get lower. And you’re not the only one in the forest. You’re not the only one racing. If you look around, people are launching much faster, at higher quality, and they keep the service up and keep improving it in production. If you don’t build that muscle, I think that’s the real cost. It’s not really that you’re missing one product opportunity. It’s really that you’re missing the opportunity to build this muscle. Without that capability, in six months or a year, when you look back, you’ll realize how much you missed and how hard it is to catch up.
How building for agents drove TinyFish’s growth (58:20)
Aakash: Talk to us about how embracing this early has helped your growth. You’ve seen this hockey-stick growth, going from 100K to 200K fast, then from 200K to 300K even faster, and you continue to see that exponential right now. How has this way of building helped drive that?
Shuhao: We see the hockey stick, and we can correlate it to a few events. But first of all, we really should consider agents as the new type of users, and there will be trillions of them. Every human will have a whole bunch of agents, and these agents are going to be the smartest users. They will look for new products. They will have their own way of testing, not listening to the product owner’s words. They will run their own benchmarks and then select. They can even check the price, calculate everything and make a decision. So we have that in mind. It means you have a really picky user, and you have to be absolutely the best choice among the competition.
What we did in the beginning that really helped us is adopting this improving in production, or you can call it hill climbing in production, where we continuously improve our service. We’re a web infra company that helps agents work with the web. There’s search and fetch, and then there are web agents. If you look at the use cases coming from the other agents calling us, they’re very diverse and versatile. We ran into all kinds of edge cases, which are impossible to fix the traditional way. Having this process running in production lets us stay ahead of the competition, and lets the agents build a reputation of “TinyFish is best in class for this particular job.” That’s why we’re seeing a lot of traffic from them.
Aakash: So you built for AI agents. That sounds like one of the key unlocks here.
Shuhao: Yeah, exactly. I think services will become more headless, and a few super agents will continue to interact with the humans, the users, and become the gateway to all the other services. So they’re actually the ones making the choices and decisions.
Where to find Shuhao (1:00:47)
Aakash: If people want to learn more, where can they find you online?
Shuhao: They can go to tinyfish.ai.
Aakash: All right, Shuhao, this has been a wealth of information. Not many people are willing to open up their actual system and break it down in so much detail to help the rest of the community. Thank you so much.
Shuhao: Thank you. This was really fun, and I look forward to the comments. I want to exchange notes with the community.
Aakash: All right, guys, drop it down below. What do you think you’re doing that he might want to experiment with? We’ll make sure he gets all of that feedback. You can reply to the email if you were listening to this on Substack. Something over 10,000 of you [listen to] every episode, by the way, so we appreciate you. We’ll see you in the next one. Have a good one.