Check out the conversation on Apple, Spotify, and YouTube.
Cold open (0:00)
Charles: It felt like after a while that the new party foul was flooding your coworker’s context window, where we all just start launching slop at one another.
Necoline: But if we could put all these learnings into one place, and so that became this Together product repository.
Pavneet: If I had to do this manually, this would have easily occupied half of my day if not more. There are 19 tickets that were filed in the last two months alone. So this is a pretty heavily requested ask. So I kind of get a pretty good sense of what the customer problem is within like five minutes instead of maybe a full day.
Aakash: Together AI just raised an $800 million funding round valuing them at $8.3 billion. I got their product team to show you their exact repo they used to automate their work. We all discovered ChatGPT in 2023 and were horrified at the output and started handwriting. What is a good PRD in the AI era?
Pavneet: I come from Amazon. I used to work at Amazon. So we used to write like 20 page PRDs. What replaces the bulk of that is actually a prototype.
Charles: We’re all going to live in a multimodel, multi-harness world. That’s essentially the new bar for UX for the kind of products we build. A lot of these things don’t get magically better with AI.
Aakash: Where does the line of product manager end and developer begin?
Why individual productivity was the wrong goal (1:44)
Aakash: Charles, so I’m fascinated by how you guys work and how you guys have been using AI. What have been some of the big unlocks for your team in productivity?
Charles: You know, Together AI is a platform for powering AI applications and agents. And so we saw, going back to the rise, back when Cursor was starting to hit its growth spurt, what an amazing superpower an AI agent could be for any kind of knowledge worker, in addition to and including product people.
But it made me think a bit about what would a world be like full of product people and engineers that could all generate as much code and content as they liked. And it wasn’t clear to me that the sum of all that was actually necessarily forward progress for us as a startup. And I imagine it’s probably the same for a lot of other organizations.
So what we set out to do as a team is ask ourselves what would it mean to use AI, but not to use AI to just make ourselves individually productive where we all just start launching slop at one another, but rather AI to make us collectively more productive, that the code and the content and the research that we did actually advanced the company as a whole.
And that led us to a bunch of decisions about what we wanted to do centrally and what we wanted to leave people to be free to do individually. And that’s largely what brought us to where we are today.
Inside the Together product repository (3:26)
Aakash: So how do you coordinate and centralize all this information?
Necoline: Hi, I’m Necoline. I’m a product manager at Together. And a few months back, we’ve been talking about the way that we use AI in the hallway, but we never really sat down and discussed how we actually do our day to day work.
And when we did, we found that a lot of people had unique workflows that made a lot of sense to the area that they were working in. Checking to see the reason behind a node failure, working with customer support, drafting reports. But then the rest of us on the day to day basis, we were doing things that were largely the same. Researching customer needs, creating PRDs or one pagers.
And so we thought, if we’re really being thoughtful about how we’re constructing these skills and how we’re thinking about agents versus skills and now, you know, loop engineering, what if we could put all these learnings into one place? And so that became this Together product repository.
It’s composed mostly of markdown files, some YAML files as well. And what we put in here are both context that is useful for us across the board. So we have this context directory and it has a number of groups of the product grouping that we have. So customer intelligence, sandboxes. But we also have the result of strategy meetings that we have on a regular basis, where we divide up product by mission, our strategy, what we want to accomplish, milestone over milestone, quarter over quarter. So that when we’re thinking through our research and we’re composing these documents with our agents, we can pull this context as it’s relevant.
And it’s also useful to be able to look at the context from other people’s sections. And so if you’re building something that is touching or going to create a joint workflow. So if I’m creating sandboxes that reinforcement learning users are going to make, I can of course sit down with the product manager from the model shaping team, or I can look at the documents that she’s already thoughtfully composed about their entry points and how they use SDKs and how an ideal integration might look, and kind of work out a really decent proposal before I then put it before her and try to do that leg work. I’m taking advantage of the work she has done on the skills front.
It’s a mix of things that are useful for anyone outside of a code repository. So everything that’s relevant to specific code really shouldn’t live here. But if you want to do something like figure out if we are serving all of the generative media models that all of the users are interested on Artificial Analysis, I can have that skill here. So I could run that as somebody who supports generative media, but someone else could as well, and they can see the output of that in the terminal.
Or things that we do on a regular basis. So this topline status one is one that I really love. At the end of every sprint we go through all the projects that we work on and we create an update of what’s been shipped, what’s ongoing, what’s coming up next. And on this skill you can just decide what areas you’re going to pull from. So for sandboxes, there’s a lot of Linear projects. But for the SDK and API, that pulls from a lot of projects. So you can pull directly from a repository, see what features got pushed, and then it’ll form an update based off of the sources that it was fed, and then you kind of edit it, add additional updates that it missed or context that would be useful for leadership.
But that’s something that used to take me anywhere from like 10 minutes to 30 minutes, depending on what I’m sussing through with the engineers. That now takes me a handful of minutes just to type in the prompts and then put the answers into the right location there.
And as you can see, this is all Claude. Originally, I think we for the most part adopted Claude Code. But a lot of the harnesses that support open source models are used to looking at Claude markdown files and being able to run them just as well. And so we’ve started to move more over to using our own models. One because they’re fantastic and it’s really nice to use our own models, and two, it’s more affordable, because Claude is, I think like many teams are learning across the globe, that Anthropic and all these other models are starting to become quite pricey.
And so this is an example of using OpenCode. It’s the same repository. This is one of the skills. It’s just a news report that I run in the morning. It looks at our competitors, other model labs, just to collect information from what happened over the last 24 hours. And I will read it over my morning coffee. It takes a minute.
So I ran this earlier, but you can see it’s going to read the skill and kind of look through all of the listed sources that I want it to find news articles for. And then it will create a nice report that I can then click into and build out or read through any information that I want to learn more about that day.
Who owns and maintains the context (8:57)
Aakash: So if we return to the context, who owns and maintains this? Because the worst thing is if you’re going to have out of date context. So let’s say Charles sets the overall product strategy. Is it his deliverable to maintain the strategy document that isn’t a good format for any AI harness? Or how do you guys manage these types of behaviors and responsibilities?
Necoline: I think for the areas that, you know, the sandboxes or the customer intelligence that I own, it would be my responsibility. And then on the general, it has been kind of updated based off of larger events. So if we have a strategy or planning session, then someone would sit down and decide who’s going to sit down and translate that into what material. So there’s typically a lot of documents that have been written in the leadup. And so it would be a matter of just translating that into a markdown file.
But the CEO will write letters or write documents of direction and that’s a very clean moment, when direction is being set. And I think that speaks a lot to how well our leadership communicates direction. You know that this is a pivot. They make it very clear, they lay it out for you, and so that’s kind of your signal.
And I think as a product team it’s just a matter of us. I tend to update this quite a bit, but I think if anyone else makes a PR it’s like any other PR. If somebody takes the initiative to create a PR or make the update to the markdown file, then that’s that. But as soon as you want to pull from it and you realize it’s missing, then you go ahead and make that change.
Where team skills end and personal skills begin (10:41)
Aakash: And going into the skills. So how do you maintain what’s going to be a team skill and an individual? Maybe somebody has a different way of writing a PRD. What is the sort of prescription or guidance? You should be using our team PRD, you should be customizing them. What’s the line?
Necoline: Yeah, I think when it comes to updating skills, there’s a rule set on where each skill should live. You want to keep it closest to the work you’re trying to do. So if it’s a skill that is referencing code, or is housed within a set of work that a team is working on that’s always in that one repository, then keeping everything colocated makes a lot of sense.
And then all the rest of the skills need to go somewhere. So you’re either going to have a personal repository or you’re going to have a shared repository, and potentially you create a branch where you test out a skill and you say, okay, first, I think this is relevant. I think this is repeatable. I’ll see if I use this a few times. And if you do, then you can push it up to main. And that’s been my approach so far. And if it’s something that I think is super niche, then it might just stay in a branch that never gets pushed.
Choosing a harness and a model (11:57)
Aakash: Makes sense. And so everybody can connect into this harness using whatever model they want, but the harness primarily is these context and skill files. Is there any other components that people need to know about?
Necoline: So there’s the repository and then there’s the harness. And the harness, there’s many open source harnesses. OpenCode is one of the primary ones that we’ve been watching as it’s evolved over time and supported a lot of the models that we support as well. But Hermes has also recently published one that I’ve enjoyed playing around with.
And as you figure out which harness you prefer in your day to day work, or if you’re looking for models that suit a specific workflow, I think a lot of product managers are starting to become a bit more familiar with what models are suited to different tasks. But as more open source models are coming out, I think it’s really interesting to see, okay, how is GLM for coding? How is it for creating these product prototypes? Or Kimi for doing analysis?
And I really enjoyed that experimentation, because as these models evolve, you feel like you’ve got a lot more control over pairing the right model for the capability that you’re after at the time.
Running feature research live (15:17)
Aakash: So, Pavneet, can you show us what this is like in action? How does someone day to day use this for customer interviews, writing PRDs, and getting things into a state that they’ll hand it over for engineers?
Pavneet: Yeah, Aakash, I’m Pavneet Ahluwalia. I’m the product manager for infrastructure at Together AI. So how it really comes into action is basically based on two processes. One is the initial discovery and aligning process, and the second part is the build and ship, which is more automated. And really what this ensures is that as a PM, when I’m shipping a PR or I’m writing some code, it’s not noise to engineering. It’s actually grounded in our best practices and architecture.
So let me show you this in action. I have my Claude Code here. I’m going to first start with a feature that I want to work on. Let’s say I want to do some research on a potential feature that I heard some customer complaining about. So one that I’m looking into right now is how many customers are complaining about shared storage resizing.
And what this feature is going to do is it’s basically linked to Pylon, which is where our support tickets live, to Linear, which is where all our project and engineering execution is tracked, and to Notion, where some of our internal product documents live. And it’s basically pulling all the information from there to show me how big is this as a customer problem, how many customers are complaining about it, give me some verbatims and some tickets that I can go and deep dive into.
What is automated and what stays human (16:57)
Aakash: So you had mentioned that certain parts of the product process are more automated than the others. Which do you feel like are more automated?
Pavneet: Yeah. So the way we see it is the part where the decisions are being made, which is defining the feature, defining the API surface area, the abstraction layer, those are very human in the loop, where I’m hands on with Claude. And the code writing part, the execution part, is actually the part which is automated.
And how we would typically do that is I would basically trigger a goal and say, give me production ready PRs for a feature to resize. So exactly the feature that we are going to work on. Resize the shared volumes for a running cluster without destroying data. Just hypothetically.
Now typically as part of this goal, I would also give it a few other tasks. For example, give me a design doc that I can review with engineering. And first, let’s verify on a PC cluster. So we would typically deploy it on a cluster just as a POC to validate everything, that it’s working as intended.
And this is where, because goals are amazing, this will spin off multiple sub agents and actually start tracking all the pieces. And it’s going to prompt me for information along the way. For example, like a PC cluster or more information about the feature. And that’s where we’ll feed it the PRD or maybe a UX prototype that I built along the way.
Aakash: So very, very powerful engineering skills. It sounds like you have a lot of confidence, and engineering has built skills that you can just give a goal command like, get a production PR ready, and it will go pull the relevant skills. They will be able to read your codebase and get something actually production ready.
Pavneet: Absolutely. We actually have a full engineering repo where it’s just our architecture details. It’s a bunch of skills which tells it, for things that we’ve done in the past, how to redo it. Like runbooks, engineering architecture design documents, which Claude has access to through our shared repo. And that’s why it’s able to build high fidelity PRs in like a first attempt really.
Aakash: Wow. I didn’t realize just how good these had gotten at AI native companies. So that really then, like you were saying, in the human in the loop parts gives you a lot more time to focus on those in depth.
Pavneet: Yeah. So if you see right now, it’s actually spun off a sub agent to look at how is the one data plane working, and then another sub agent to check how is the control plane working, and now a new one to look at docs. Is there existing internal or external documentation on how we might do this?
The PRD writer that interviews you (19:56)
Aakash: And do you have a point of view? You’re using Claude Code in the terminal. What’s the easiest way to use a shared team repo harness?
Pavneet: Yeah, so actually if you look at this, it’s currently running in my OS directory, which has actually pulled a lot of the shared repo artifacts. So for example, Neco shared a few artifacts around research. So it’s actually this skill, the feature research skill, which is actually a shared skill that everybody uses in the team.
Then there’s another one which is my personal favorite, which is PR writer. This is actually pretty cool. Let me just actually trigger it off. So, resize shared volumes for existing running clusters.
So what this actually does is it does a turn by turn interview. It’ll interview me basically on what are the decisions I want to make in this feature. And if I show you the skill, this is the actual skill. It ingests, it can take a prototype. So if I already have a UX prototype it’ll ingest that. If I have a running POC, it can take that. Or nothing, just a simple prompt.
And then it has access to all the information, the shared context that Neco shared about. It’s pulling all of that in, and then it’s going to ask me questions turn by turn. And it’s specifically instructed to push me, to challenge me on my assumptions and basically the decisions I’m making. And then it goes into drafting a PRD. And we’ve given it a question bank basically to ground it in what type of questions typically we need to make. For example, what are the trade offs? Is there any one way door decision we are making along the way?
Aakash: Amazing. So we’ve spun off, I think we have currently three agents working for us, right? Research, PR and PRD.
Pavneet: Yeah, so this one is actually done. So the research agent came back. It’s saying all these customers have asked for this feature in the last six months. There are 19 tickets that were filed in the last two months alone. So this is a pretty heavily requested ask. I have the source, so I can actually go into these tickets if I want to dig deep, exactly what happened and why did customers ask this.
Aakash: And what was Pylon again?
Pavneet: Pylon is where our support tickets live. It’s the support platform.
And then actually it figured out that we had started working on this feature, because it has access to Linear as well, and it was partially implemented but it’s broken in some way. So we stopped working on it. So now I’m not duplicating, I’m not replicating work. I can actually build off somebody else’s work, or actually talk to that engineer and figure out what happened, what issues we ran into.
Aakash: And is this hooked up to the live codebase, so it could go check in case there’s some discrepancy between Linear and the codebase?
Pavneet: Yes, because it’s in my personal OS directory, it has a skill which gives it access to all the internal GitHub repos and it can map that.
By the way, Aakash, later on I can show you a god’s eye view of what this looks like across all products, across all repos, as well as across all customer feedback.
Half a day of research in five minutes (23:09)
Aakash: Okay, I’m excited for that. So beneath here we’ve got the research report. What’s your take on this research report? How long would this have taken you? What do you give this out of one to 10?
Pavneet: Yeah. So if I had to do this manually, this would have easily occupied half of my day, if not more. And this is actually pretty good. It gives me the brief snapshot that I wanted to know, which tells me the feature passes the smell test. There are enough customers asking for it. I know most customers actually want to increase their storage volume. So I actually know what’s my MVP use case.
And then I have a source, so I can actually dig into some of them to actually get a better sense of the customer pain. And it gives me verbatims as well, which kind of helps. So I know for example here it calls out a customer, and we have a separate tool called customer insights where all customer calls live.
So let me go into that. This is the customer insights tool, where every customer call that our sales team is getting into, all our support tickets, are getting summarized and cataloged. So I can actually go into this and actually see what exactly happened with Moon Valley. So they wanted a 50 terabyte, but there’s a lock icon in the UX which is preventing the action. And they reached out to the sales team and they basically want a self serve workflow.
So I kind of get a pretty good sense of what the customer problem is within like five minutes instead of maybe a full day.
The customer insights tool (24:25)
Aakash: And this customer insights tool. So you guys have built an MCP server sitting on top of Gong, and this has been visualized by v0. That’s what we’re looking at right now.
Pavneet: Exactly. It covers both Gong, Slack, Pylon. At least those. So it’s actually more than just Gong.
Aakash: And who maintains this? Sales ops?
Pavneet: You want to speak to that? It’s largely automated.
Hassan: Yes. Me and my team maintain it. So my name is Hassan. I run the developer experience team here at Together. And me and my team have built a bunch of this kind of stuff. We maintain and we continue to add new features, but it’s largely kind of working by itself at this point.
We have like a daily cron job that runs every day that grabs all of the calls, all of the Pylon tickets, all of the Slack channels, everything that happened in the last 24 hours, and it adds it to the database that this is running on. And what Pavneet is showing is the daily screen where we show like five to 10 kind of insights from customers every single day based on all these calls. We also have an MCP server. We have a chat where you can ask it anything there. There’s a lot of facets to this tool as well. But yeah, my team continues to maintain it and add new features here and there.
Aakash: Okay, so developer experience maintains various MCPs. What other MCPs are you maintaining?
Hassan: Good question. We have this one. I mean, a lot of these tools are exposed differently. This one has an MCP server. For example, another tool, it’s called Orchestrator. Charles is going to show it off and talk about it momentarily. That one has kind of just a UI and is a wrapper on top of all our GitHub repos.
And really my team just experiments with a lot of this stuff. We build things that we think may be useful for the product team and the rest of the company. And some of the internal tools kind of flop and we’re like, well, this is not used very much. And probably some of our most used ones have been this customer insights MCP and app, and Orchestrator, and agent evals which I’ll also show off. So those are like the top three that people have been using.
Aakash: Sweet. So we get to see all three. Awesome. And Pavneet, how’s our PRD and our PR agents looking?
Pavneet: Yeah, so it’s now asking for evidence. So this is where typically I would run this in a single Claude chat, because this is sort of somewhat sequential work. But typically then I would take the output of this research, which is pretty detailed for a starting point, and I would actually give it to the PR agent to run with. So now it has a sense of the customer problem. So it can ask me more educated questions on, hey, what are the trade offs? Should we build as an abstraction? Should it be self serve? When do we pull in support? Those kind of questions.
Aakash: So almost like managing different agents that feed each other.
Pavneet: Okay, so now it’s asking me, do you want to give me a Figma or a running POC or some screenshots? Or I can just say no, run with it, let’s do it turn by turn by discussing it.
What a good PRD looks like now (27:31)
Aakash: We’ve all gone through phases of using AI for PRDs. We all discovered ChatGPT in 2023 and were horrified at the output and started handwriting. Then we discovered Claude and said, okay, maybe it can write well. And then it almost feels like people were overloading our colleagues’ context windows with long documents. What is a good PRD in the AI era?
Pavneet: Yeah. So that’s actually a question we thought a lot about. Historically, companies started using PRDs as a gating document, where all stakeholders need to come together, align on it, and only then we’ll start building the feature.
We see it a little bit differently, where it’s a tool to trigger ideation and problem solving. That’s it. And it’s short. It’s usually one to two pages. It defines the customer problem well so that everybody has a shared understanding of the customer problem we are trying to solve. It lays out some solution options. Doesn’t have to be fully thought through necessarily, but at least what are the different ways we could go solve this problem. And then a sample user journey. Based on the option we want to go with, it’ll define either an API based journey or a UX based journey, to show these are the three, four steps that a user would go through in the solved world once this feature is shipped.
And that’s it. That’s enough to actually have a detailed discussion around, how should we build this feature? What should engineering design look like? Is it even worth building or not? And that’s what the PRD template that this skill uses builds off.
And actually what replaces our typical six pager, like I come from Amazon, I used to work at Amazon, so we used to write like 20 page PRDs. What replaces the bulk of that is actually a prototype. So we built a separate skill. Let me start a new Claude window.
And there’s a separate skill called UX prototype, which you give it that one pager that we just discussed and it’ll give you a very detailed prompt on how to build it in something like Figma Make or any design vibe code tool. You can use it in v0, you could use it in Claude Design, or even Figma.
And that’s where a lot more debate will happen, because now everybody can visualize the solution, see how it’s going to interact with the user. And that’s where we find a lot of the best critical feedback comes from, from engineering, marketing or anybody else.
Aakash: Do you find you’re living outside of Claude Design then a lot, you’re getting this to generate the prompt and then giving it to Figma Make?
Pavneet: Yeah, the only reason I use Figma Make is because it’s easy to share a Figma Make link and others can iterate on top of it. It’s easy to click around. Claude builds HTML files, which it’s hard to keep version tracking and stuff like that. Figma is just more shareable. That’s why we use Figma Make.
Aakash: Makes sense. So how do our PRD and PR agents look?
Pavneet: Yeah. So now it’s asking me questions. So it’s saying, the evidence splits into two different directions. One is multi-tenant substrate quota bumps and the other is dedicated conversion work expansion. Which one do you want to go for?
I actually want to go in a very specific direction, just looking at the feedback we got from customers. I want to say, we should focus on increasing the attached storage volume sizes for tenants. Tenants here is basically a customer cluster. So, tenants in a running cluster.
Aakash: So here’s where your domain expertise, your human judgment is coming in.
Pavneet: Yeah, it’s not a substitute for me, but it really does two things. A, it speeds up the whole PRD writing. It solves the blank page problem to a large extent. But it also catches a lot of things that I might not have thought of. I often find it’s asking me questions that I might have missed if I’m not spending a lot of time thinking about the problem.
So, ignore capacity issues. I’m just giving it a sample to focus its energy and to reduce the scope so it’s not trying to boil the ocean in the problem space.
Aakash: So a 10 out of 10 that comes out of it is going to be that very short document that accompanies a prototype, if this all goes well.
Pavneet: Yeah. So I actually did this and I have that PRD. If you want I can show it to you what that looks like.
Aakash: Yeah. Let’s take a look.
Reading the finished PRD (32:02)
Pavneet: So this is what the final output looks like. So first of all, it’s telling, where’s the evidence? It looked, we had 20 Pylon tickets across 14 customers. So straight up, there’s evidence.
It first defines the customer and the business problem, like we talked about, the 20 confirmed customer issues. It’s name dropping some customers just to give us a sense of which customers. Is it our top big customers? Is it a long tail of customers? And then it gives you a sense of what is the actual pain. It seems like we cannot update this in the UI, and that’s basically the ask, they just want to self serve the volume resizing. And that’s pretty obvious from the customer verbatims.
And then you go into goals and non goals. This is important for scope creep, which is the number one problem I feel like PMs and engineering face. So it gives you some goals. It’s giving me a goal that, if done well, this should handle over 90% of resize requests. It works in both directions, upwards and downwards, scale up and scale down. Billing should adjust immediately.
See, this is something I did not think of actually. When somebody reduces the size or increases the size, we need to make sure that the billing matches up and they’re not over or undercharged for it. And capacity block request. What happens when we don’t have enough capacity? If it’s a fully self served experience, when do we actually need to get support involved? And then it defines some non goals as well.
Okay, user stories, so that whoever is reading this, when we are doing a discussion in engineering, everybody understands the key use cases. And then the proposal overview. It’s giving me some API specs. It actually gives a detailed API design as well. So we’re going to add to our existing API some changes on how to allow for resizing. This is the user story, what are the different API specs and what are the API calls. And then UX as well, end to end user flow, happy path, what are the steps that a user will follow.
Aakash: So I feel like it always gets the headers right, but sometimes the devil is in the details. Is it nailing the details?
Pavneet: So let’s look at this one. Admin sets a new size in the console. API validates. This is some internal detail actually. So maybe this part not necessarily needed in the user flow. This is actually internal detail. And then it shows the status. It shows that it’s available and billing rate is updated. So it got the user journey right, but it bled some internal detail into it as well.
Aakash: So it’s still going to need a little bit of editing. You can’t just immediately take this and start sharing this with your colleagues.
Pavneet: Yeah, absolutely. Actually, at the top it says this is a draft one pager PRD. It’s not meant to go wide distribution for everyone. And that again goes back to the process that we were talking about, where in this initial discovery and design phase, it’s human in the loop. So the expectation is that the PM will go and make changes, maybe make some changes to the API, maybe change the scope a little bit, before we circulate this widely.
Orchestrator, a view across every repo (35:07)
Aakash: Very cool. So we’ve got the repo, we’ve gotten to see it in action. We got a little preview about an orchestrator agent. Charles, can you walk us through this? How do you get to see this bird’s eye view at the top of their product ladder?
Charles: So we built this great internal tool which we call Orchestrator. The basic idea is each of the individual product people are sort of married up hand and glove with their corresponding engineering team. And Pavneet gave you a good example of how that works, where he wants to be able to go pretty far into the definition and implementation for some of the things he works on, and the specific engineering team he works with has given him the kind of skills connected to their codebase that he can go do that.
From my vantage point, what I need to do is be able to get a check on where things sit in all kinds of different things that we’re building as a company. And I don’t want to have to build the equivalent of Pavneet and Hassan and Neco’s environment.
So I have this nice tool here called Orchestrator, and you can see that essentially all the major repositories we have as a company for all of our products are represented here. I’ll give you a quick example. Let me find Pavneet’s t-cloud. So you can see here we have the repo that Pavneet was just living in, t-cloud, but you can see that it coexists along with many other repos.
So for sake of argument, one of the other products we have we call model shaping, which is basically the ability to adapt the behaviors of open weight models. And I can decide how I want to interrogate what’s going on in that codebase, what’s going on in that product area. I can pick my harness. I can use either OpenCode, OpenCode Claude, or Cursor. I can pick my model. So in this case, I tend to be a GLM OpenCode kind of guy.
And I could ask some question like, what’s the most recent model we have enabled for supervised fine tuning? We basically have more than 30 models that you can adapt and fine tune, but these are changing all the time. I don’t want to have to go understand the entire world of the product lead for model shaping. I just want to understand what’s going on in this one specific area, to answer this one specific question.
And you can see that essentially what we’ll do in this case is we’ll actually create a sandbox. We’ll clone in the repo. All this happening in the background. And in a minute it’s going to basically interrogate that portion of the codebase, and it’s going to go research what are the most recent changes and let me know what happened most recently.
You can see it kind of cogitating right now, like the proverbial cooking show. I have kind of a synopsized version of the conclusion. And in this case it looks like the answer is the last model that we enabled for supervised fine tuning is the new NVIDIA Nemotron Super 12B model.
So I can do this exercise essentially across any product, any portion of the codebase for that product. And it’s not just limited here, I’m just asking questions. But if I also find some part of the product that annoys me and it’s something small, I can actually generate a pull request from here as well.
Aakash: Okay. So you’re mainly living in the Orchestrator, not inside Claude Code and the team repo.
Charles: I use Claude Code and the team repo if there’s some requirement I’m writing myself for some part of the codebase, then I would do that. But if it’s me shopping across all the different products we have and it’s some small UX change or things like this, then I would sooner use the Orchestrator.
Because this is basically not just pointed at all the different repos we have as a company, but it’s inheriting all of the skills and MCP servers that are local to each of those repos. So I don’t want to have to build all of that into my local OpenCode just to make one small change. It’s a lot simpler to just use Orchestrator, and I know it’s going to have the latest greatest skills and MCP servers for that portion of the codebase.
Aakash: Got it. So who are the other users of Orchestrator within the company? Who is this product exactly built for?
Charles: It’s sort of intended for casuals. It’s intended for, let’s say Pavneet wants to investigate something in inference, or let’s say Neco wants to make a suggestion on our infrastructure as a service. They don’t want to build and replicate all of the specific skills and context native to that particular portion of the codebase. It’s a lot simpler to just use Orchestrator where everything is kind of maintained server side.
Aakash: Okay. So this is for whenever you’re casually working, maybe trying to learn something about another team. This is also creating that connectivity.
Shared context is a hierarchy, not a flat pool (40:24)
Charles: Yeah. One of the things that we talked about as a team was, it goes back to this point about how much is shared and how much is individual. And there was a point at which I thought, well, wouldn’t it be great if everybody knew what everybody else was doing? And there was sort of like we all had the same context and everybody else at all times.
And in reality, most people don’t have a whole lot of motivation to want to understand all the depth and nuance of the context of somebody else’s area. They want to know just the amount they need to get one single question answered or to get one single problem resolved, and they’re really not interested in the rest.
So we sort of set aside the idea that there was one big broad flat set of context that we’re all going to swim in, and it’s much more like a context hierarchy. And some of us belong all the way down to the bottom of the depths of that hierarchy and some just want to traverse the top.
How to build your own orchestrator (41:20)
Aakash: Fascinating. Okay. So if a team or company watching this wanted to replicate what’s being built in Orchestrator, how are they going to spin up their own version of this?
Charles: Yeah. So from what we see in terms of our own customers, this is becoming increasingly common. I was at an AI conference in Paris the other week and I saw a digital services and marketing company called Process, and they basically already built their equivalent of this same thing, where they have every portion of the codebase you want to change all in one place, everything can be spun up as a sandbox and generate its own unique pull request or its own questions.
So I think this set of tools is not that out of reach for most software development organizations. And this idea of, we’re all going to live in a multimodel, multi-harness world, and the main endeavor then is how do you organize shared context? That’s kind of I think what everybody is starting to build.
To give a more specific answer, Hassan, what was the total time invested would you say to build Orchestrator?
Hassan: I would say a few weeks of work, maybe a month of work, to build it, of engineering time. And a lot of it was kind of just trying to figure stuff out fairly early, trying to figure out the right architecture, the right tools to use, a lot of that stuff. But as Charles said, I think this has kind of been replicated by a bunch of other companies. There’s like a few open source versions of this as well that exist out there. So it’s kind of easier than ever once you have the architecture. But yeah, for our team it took a few weeks.
Aakash: So if we’re just drafting the PRD for an internal orchestrator for yourself, what are the key things?
Charles: It needs to hook into all of your codebases and your different repos. It needs to inherit all the active MCPs and skills from the different team repos.
It needs a sandboxing mechanism, because essentially each individual thing you’re researching or each pull request you’re going to propose, that all gets done in a sandbox environment. So you’re basically cloning a fraction of the repo, the portion of the repo that’s necessary in the sandbox, to generate that one pull request.
You’re also going to need some form of a model gateway or router. So you notice I had a whole choice of models that I could use. So that’s typically another piece of the puzzle.
But as Hassan mentioned, every one of these components I’ve mentioned so far, either it’s relatively quick to build, there are open source tools and libraries that let you do these things, and there’s commercial shrink wrap software if you don’t feel like doing either of the previous two things.
Agent evals, testing the product as an agent (44:19)
Aakash: Fascinating. So we’re walking through the whole product development life cycle. The last step, evaluating how these things are going live. Hassan, how are you guys doing agent evaluations?
Hassan: Great question. As you said, this is kind of the final step that we do. I’m going to go over this tool that our team built called agent evals.
And just to give a little bit of context on agent evals. I think the whole world is moving to this agent centric way of doing things. You don’t kind of manually write code anymore, and it’s moving up the stack with, you just give your agents stuff to do.
And so my team is called developer experience. I think Charles is this close to renaming it agent experience, because it’s something we’re increasingly thinking more about and it’s very top of mind. We’ve rearchitected our docs in this agent first way, where it works very well for humans obviously, but also for agents. On some pages on our docs, when an agent is reading them, we’ll inject something extra if we think it’ll help the agent. We worked on the Together MCP server, we worked on Together skills. And so we’re exploring all of these ways that we can make it as easy as possible for agents to use our product.
And agent evals is kind of the tool that ties them all together, or the tool where we can actually say, okay, we’re pretty confident in how this works.
Okay. So agent evals, there’s a lot here. Agent evals is the tool we use kind of at the end of this product development life cycle. We have the product, it’s shipped. Some major feature or a product or second version of the product, and now we need to validate that it actually works well with agents.
So what we do is, I tend to work with a lot of the product managers in each of the different areas, and we write a series of tests of the main ways that we think users will use a specific product.
So let’s take fine tuning as an example. If I click on fine tuning, this is a prompt that we give. This is one fine tuning task that we’re testing. So here we’re saying, hey, we gave it a data set and we said, run a fine tuning job for this data set, and when it’s done, spin up this new fine tuned model as a dedicated endpoint and evaluate it, where it’s going to send it some inference requests, and then delete the endpoint when you’re done.
And this actually touches on a few of our different products. It touches on our fine tuning product and our inference product. And so some of these tasks are a little more simple, some of them are a little more complex. But the point is that the most important part is to define a series of tasks and give it to this tool.
And what we do actually has a very similar architecture to Orchestrator, which we talked about. So what it’ll do is it’ll spin up a sandbox. It’ll spin up right now Claude Code, and it’ll give it this prompt. It will give it a Together API key and then it gives it whatever it needs for the prompt. In this case, we gave it a data set. And then it just watches Claude Code do its work. It lets it do its thing and then it evaluates it at the end.
And for us, this is like the best test of, can an agent actually do stuff in your product. We try to make everything in our product so that you can do everything in our product, for example in our UI as a human, but also as an agent through our API or CLI or SDK. So in this case it actually got it correct.
But we go very deep here. So we run a lot of different runs. Some of these runs are just using our docs as context. Some of them use our MCP server. Some of them use our skills. And we can click into every one of these and we can get a lot of information on the run, how it worked. We can get some high level info. We can get improvements, which is one of the most important pieces of this.
And this is something we see a lot. We’ll ship a feature, or in the validation stages of a feature, and we give it to agent evals and then it’s like, oh well, I had trouble doing this thing. For example here I said it wasn’t able to discover what fine tunable models we have. Obviously this is a big problem if it can’t find this. And it said, it couldn’t find a docs page listing currently fine tunable models, and so we should do this. And so we make a ton of improvements to our docs based on this.
In this case actually what happened is we had a page that listed all the fine tunable models, but the agent actually couldn’t find it. We were able to go into the transcript. You can see the full transcript of this agent and what it did. We gave it this initial prompt and you can see all of the different turns. We can see that the agent started exploring the repo. It started looking at the data set. It did all of this stuff. The full transcript.
And we looked at it and we saw that actually it had trouble finding that one page in the docs, because it’s not linked in our main fine tuning quick start. So then we’re like, okay, well, we’ll just go submit a PR to our docs and add that page to our fine tuning quick start.
So agent evals has been responsible for like dozens of docs fixes that we’ve done. And it’s been really valuable to get this bird’s eye view from agents on how good are agents at using this particular tool or this particular API that we just shipped.
Validating design immediately and continuously (49:41)
Charles: And Aakash, I just want to add on. I mean, you think about any sizable feature or product that you’d build in the past, and the idea about, well, did you get the design right? Can the outside user, whether it’s someone technical like a developer or non-technical, be consistently successful using that feature?
And think of how long you had to wait to validate that as a product person before. You’d have to wait till you did a bunch of user tests, and then you’re going to get a bunch of conflicting signal, or you’re going to have a bunch of developers evaluate it and they’re all going to say, well, I don’t think this is sufficiently Pythonic, or I don’t like the way you did this syntactically. And so it’s really hard to get timely signal and it’s really hard to get objective signal.
And here, essentially at this point, for a lot of our products agents are already the majority user. And so the nice part about that is, did the feature work from a design point of view? This is something that we can validate immediately and continuously.
So it’s really powerful to always know where you’re at, and it’s really powerful to have the confidence to know that whatever your documentation says your product is, you can be sure that it actually is that, because essentially every few hours we have agents revalidating that. And across time, what we plan to do is keep expanding the range of harnesses and models that we do this for, because that’s essentially the new bar for UX for the kind of products we build.
Aakash: Okay, very cool. So if your product isn’t used by agents yet much, does this still have value? Is this able to simulate what a human is like? And should people still be setting this up?
Charles: For us it is, because our users are developers and so they’re going to use our SDK, and this is basically validating that there are valid successful paths using our SDK.
If we had a more GUI, web front end intensive product, we would have to adapt this tool to use more of a visual reasoning model that could actually do the clickthroughs and interpret the screens and see whether or not that was naturally intuitive to the model. So it could be extended to human centric examples. But for our case where agent use is already so popular, it works as is.
Where the product manager ends and the developer begins (52:26)
Aakash: All right. So we’ve been able to cover front to back from customer research through to actually evaluating how agents would be using your feature. This is kind of the whole product development life cycle. I’m curious where this ends. We kind of drew the line here at create the PRs, create the prototype. Where does the line of product manager end and developer begin?
Charles: I want to think how to best respond to that without just repeating what Pavneet just said at the beginning of our whole thing.
I think that the essence of what the product person does versus the essence of what the engineer does, ironically, is probably not all that different than what it was before AI, in the sense that the most valuable thing the product person can do is bring a unique insight about the market that’s been well validated by lots of internal and external context. And that’s essentially what you still saw Pavneet do.
And the essence of what an engineer does to add value to the company is to arrive at a design that is the most efficient way to meet a need. The essence of the engineer’s job is to arrive at a design that most efficiently satisfies a need, and is also something that you can maintain and extend, and contributes to the long term architectural strength of the product.
So the essence of each thing I don’t think is actually all that different today than it was before AI. What’s different is the convenience with which the product person can reach into the engineering world and accomplish small and medium sized tasks, and the inverse is true as well. The degree to which the engineer can reach into the product management process and answer their own questions.
So just like it’s possible for me to interrogate the codebase and make small pull requests, it’s just as possible for one of my colleagues in engineering to use that customer insights tool and do their own analysis and have their own standing query for whatever the customers have been asking for in the past month in their area.
So it’s sort of easier for each side to reach into each other’s area to do small things. But in terms of what each person’s supposed to contribute that brings their unique talents and perspectives, I think that’s actually the same core as it’s ever been.
What it actually cost, and what it did not fix (55:10)
Aakash: Amazing. So I’ve been preaching to people, get your PM OS, get your team OS, get your company OS. You guys just demonstrated you’re actually living that reality. So you’re at the very top of the AI adoption curve.
And I heard a really interesting observation from Chamath Palihapitiya. He said our token costs are something like doubling every 70 days, but our actual engineering productivity is just up like 5%. When you look at it from a product lens on that, how are your costs on AI growing, and are you seeing some sort of tangible productivity gain that you can point to from it?
Charles: Yeah. So we never were in a place where we were doing story points or other kinds of velocity measures. So I can’t prove it on that level, but I would definitely say that our velocity has gained more than 5%.
I find claims of 3x to be very suspicious. When you get a team of let’s say a dozen engineers and a product manager, there’s so much of building software which is discovery, debate, re-evaluation, coordination. And a lot of these things don’t get magically better with AI. So even if you compress the research and even if you compress the coding and the testing, I think that’s worth a lot in terms of velocity. But I don’t know that I would say our experience has been that 3x or some huge multiple like this is the case.
And as far as expense goes, I think we went through the same surge that a lot of folks did. It was easier for us to mitigate because we can use our own open weight models and they’re a lot less expensive.
But the other part is, it goes back to this first point, which is I think if you’re focused on making teams collectively productive as opposed to yourself feeling individually productive by producing lots of output, I think it’s unlikely that you wind up with these crazy token budgets of three times people’s salaries or things like this. I don’t know that we ever reached that type of peak.
Closing (57:28)
Aakash: I like how measured you guys were in selling the benefits of all of this, where none of it was overhyped, but we got to tactically see how it helped both individual IC PMs like Pavneet and a product leader like yourself, Charles. All four of you guys, thank you so much for dropping so much insider knowledge about how Together works.
Charles: Thank you for having us!