AI 開発の加速に伴う ChatGPT の高速維持に関する発表

InfoQ Homepage

Presentations

Keeping ChatGPT Fast as AI Development Accelerates

View Presentation

Speed:

50:14

/presentations/openai-performance-engineering-agentic-coding/en/slides/slide-1785151851732.jpg)

まとめ

Martin Spier explains how agentic workflows dramatically increase code change volume at OpenAI. He discusses the hidden systemic performance costs of rapid shipping beyond GPUs, and shares how deploying always-on AI agents automates profiling, regression detection, and continuous optimization to maintain product speed and scalability at massive global scale.

Bio

Martin Spier leads ChatGPT Performance at OpenAI, where he works on making ChatGPT faster, more reliable, and easier for engineering teams to operate at scale. His work spans AI and cloud infrastructure, performance engineering, observability, reliability, platform engineering, and developer productivity.

About the conference

QCon AI is a practitioner-led event focused entirely on the engineering discipline required to scale these workloads safely. It provides direct

access to the architectural playbooks and failure metrics that peer organizations use in production.

INFOQ EVENTS

• August 27th, 2026, 1 PM EDT

Below the Framework: Why Agent Context Is an Infrastructure Problem

Presented by: Boyd Stowe - Founding Solutions Architect at Tacnode

Transcript

Martin Spier: Today I want to tell you guys a story about two accelerations that are happening somewhat at the same time. First one is around growth, how we're actually growing user base faster than ever. Not just OpenAI, ChatGPT, but in general companies are reaching their first few million users a lot faster than before. The second one is how our development workflows are changing given agentic coding and all those things, and how that is changing the rate of shipping of things we're getting out of the door, and how that is actually affecting how we keep our products fast, efficient, so overall performance engineering.

My name is Martin. I lead the ChatGPT performance team at OpenAI. Almost my whole technical career was in performance engineering. I think I was doing that for 16 years or so, or more. The bulk of that time was at Netflix, where I spent, I think, almost 9 years. Dio mentioned the early cloud days, that's around when I joined, when people thought, will I actually leave my data in the cloud? That's like magical thing there. It feels like the moment is similar today. I spent some time at Snowflake, Expedia, and a few other companies doing that as well. Prior to OpenAI, I was with another AI company called Parasail. We did inference as a service. I was leading the whole engineering team there. Prior to that, I was at a fintech out of Brazil called PicPay, where I was leading whole infrastructure platform and developer experience. As you can imagine, my whole career is really in these central organizations. It's an area I really like. I call it the plumbing of the house. No one sees that it's there, but once it blocks and it breaks, all hell breaks loose. It's always a fun part of the company to be at.

Growth Pace

Three things changed. The first one is growth pace. Not just OpenAI, like I mentioned. Companies are reaching their first million, first 10 million, first 100 million, maybe more users a lot faster than before. The curves are exponential, but a lot faster than before. Also, we're shipping more code and more things out. We're getting more things out of the door with agentic development. There's probably some talk here around AI slop and everything that's going on. We are shipping more logic out the door. The third thing is, previously, we always had this assumption that there was a human that understood all the changes that were going out. It's like someone actually coded that and architected it and know what's going on before they decide to push something out. That is not entirely true anymore, at least from my point of view today. I know there might be a slight bubble, but the abstraction layer is a bit higher.

People are delegating more things to the agents, so they don't know the details of everything that might be going out the door. It's something we need to keep in mind. Of course, with all of that, it puts pressure not just on performance. Performance is one of those problems, but everything else that goes around it. Anyone from GitHub, GitLab here, or anything like that? It's putting a lot of pressure on that infrastructure, but performance too. We're just using more and more resources. We're adding logic that adds latency. It introduces a lot of challenges, and we need to keep up. We need to evolve our practices so we can keep up with that fast pace of growth and the fast pace of development.

Going into ChatGPT, a bit of the history, just to give you a bit of context about the scale I'm talking about. ChatGPT was not launched in late 2022, as a full-fledged consumer-based application that was built from scratch to support hundreds of millions of users. It was launched as a research preview. Here it is. It's pretty cool. You guys should try it out. Quickly, users started coming. Once you have those users coming and coming pretty fast, that research preview becomes a product, and product has requirements. You have latency requirements. You have reliability requirements. You have support requirements and everything that goes behind the scenes to make that work for a large volume of users, especially if they're relying on that for their day-to-day on the things that they need to do. The growth was quite impressive. I think very few companies can tell that they reached their first million users in five days from launch in a research preview.

This is pretty cool. We have a million users here. You can imagine how behind the scenes the team supporting the infrastructure were at those times. Incredible growth, incredible for the product, a lot of pressure on everyone behind the scenes. That growth, it did not stop. It did not slow down much. ChatGPT continued to grow and it's still growing pretty fast. The last official number I can share is from February this year, I believe, when we reached 900 million weekly active users. That's 900 million people using ChatGPT every week. Just to let it sink in, that's almost 11% of the whole human population. It's a lot of people. As you can imagine, chat, it's not the lightest thing to do compute-wise. There's a lot that is necessary on the back to make that happen. That growth was not smooth. It was not just adding a few million users every day and keeps getting pretty predictable.

It was full of spikes, as you can imagine. We were launching products all the time. Some of those products became quite viral. Huge spikes on users, and that, of course, drove adoption. Just this one, you might remember the image generation spike from last year. Maybe you generated one of those images that I cannot say the name right now. Within just the seven first days, over 700 million images were generated by over 130 million users. Image generation, pretty quick. That was putting a lot of pressure on the teams behind the scenes, scrambling. This is the thing that's quite hard to predict when something like that becomes hugely popular. Again, it's something we want. Everyone wants their product to grow. This is really great.

Chat is not used only in the U.S. It's used all across the globe. With that, if you support infrastructure, architecture, all those things, you know that adds complexity to things. Like, where are my POPs? Where are my actual application servers? In our case, where are our GPUs? Are they closely together? Do we have enough capacity in all those regions? Or, how is the network connectivity between all those different data centers? The architecture, where is data? How is data being replicated? Where is data located, given all the different constraints you have across the globe? It adds a lot of complexity to the architecture to support a good experience, or to be able to do and provide a really good experience to all our users across the globe. It was not just growth, but the product did not stand still either. It wasn't just a launch or just users joining.

There was a lot of change beyond all just the model changes. The models were extremely important, but there were a lot of launches along those four years, less than four years now. There were GPTs that were voice. There were agents, all different things, image generation, a lot of product launches, some a lot more viral than others. The product was changing constantly. A lot more things, different workloads that had to be supported, more complex architectures behind it to support all of that. The product was changing while it was growing really fast as well.

Change Velocity

This is, I wouldn't call it new, but it definitely accelerated more since late last year. Especially since the launch of Codex, our shipping rate changed dramatically. I'm not discussing value or anything like that. That's a way deeper discussion, but it's just the volume of change entering our systems increased a lot. When everyone thinks about agentic development, the first thing that comes to mind is automating the proper coding tasks, like develop this function or that application, or refactor this, refactor that, and reduce the overall time it takes to get that done. Shorten the time to get from an idea to a PR that gets merged to production. The part that really surprised me is not just that, especially with newer models where the agents are a lot more capable, they can take more complex tasks. We can delegate things that will run for a lot longer.

Developers started going multi-threaded. They started actually not working on a single thing at a time. Developers are working with maybe 7, 8, 9, 10 different things at the same time. They keep on blocking them as things go. I don't know how much you're coding these days with agents. You're probably hearing the pings saying, I need the user input right now. They have multiple development boxes, or maybe you're working with org trees. In general, a lot more parallel things going on. It's not just faster to ship, but shipping more things in parallel.

Some of you might be familiar with this benchmark from DX, where basically they study a few companies, got some numbers about the volume of PRs that get shipped, that merge per engineer per week. Then you split that into different size of companies and the type of company. As you can expect, smaller tech companies are shipping more PRs, they're getting more things out. At p90, you're almost at five PRs a week per engineer, which is quite fast. We are a lot faster than that. We're shipping a lot more these days. Unfortunately, the latest number I can share is from late last year, so October, November. If you've been following, you know there was quite a huge development since late last year in coding agents. Even back then, back in October '25, the volume of PRs that our engineers are shipping every week increased by 70%. Almost every engineer at the company today is using Codex on a weekly or probably a daily basis.

Every single PR gets automatically reviewed by Codex. It's not just the coding tasks. Like I mentioned, Codex became the workbench to do everything you need to do during that whole development workflow. It's not just coding, but I found a bug in production, go troubleshoot it. Or, I want to understand a metric, a regression, go talk with the observability tools and fetch the metrics and explain to you what's going on. Or to do data science and data analysis, I use Codex to do that as well. All the other boring things like, please summarize that Slack thread for me because I don't want to read it all. A lot of productivity tasks that just make your day-to-day a lot easier. Not just engineers. I manage a team. I probably don't spend too much time coding these days anymore, but I use Codex for everything. To prepare my one-on-ones, to summarize and write documents. Even this presentation, I started it on Codex to go over the outline and to discuss what I actually wanted to present and how to slice that, and all the images that got generated, all the visuals in this presentation. It's being used for everything on your day-to-day, not just your pure coding tasks.

With that evolving of the software engineering workflow, we're shipping things faster. It takes less time for things to get out. We're working on more things in parallel. We're delegating more. That agentic development increased throughput and parallelism. That increased the volume of changes going into our system. I'm not saying value, feature, I think just changes, things that are going out. It might be a feature, might be a bug fix, might be a refactor, might be something else, might be a config thing, but changes going out. Shipping all those things, every new feature, every new line of code, it has a hidden cost. Ideally, that hidden cost adds value. It's something you want. It's something you want out. It's a new feature, a bug fix, but it has a cost somewhere. The cost might be tiny initially. It might be invisible. You might not notice it in the first place.

That increased rate of change just exacerbates that. It's nothing new, but it's happening a bit faster. All that change going out, all those new extra if conditional statements, every extra network request you're making, every data structure that you decide to hold in memory, it's consuming from a shared budget. Your latency, it can get so bad to a certain extent until your users decide to stop. If you're adding logic, you're doing more things, you're adding more checks, whatever you're doing, that always adds a little bit of latency just because you're doing a bit more. It just expands to the point where, this experience is not good, and I don't want to use your product anymore. Same goes for hardware resources. You only have a finite amount of memory. You only have a finite amount of RAM. Everyone will raise, cloud can expand forever. No, it cannot.

It's not as elastic as you think. It just makes things a bit less efficient. It's consuming from a shared budget. I'm not talking purely about those big regressions, like things that you clearly see from one deploy before and after that something drastic changed. I'm talking about even small things. Like I said, that small extra if statement that you added to check something that is important. Small things that on a database, they don't make a difference. There's no difference at all when you're comparing before and after. All of that compounds. You keep building up a little bit, little bit, little bit, until it becomes a bigger problem. It becomes a concern. You might be running out of headroom. It might be making your latencies too bad for users to have a pretty good experience in the app. It just compounds. With that increased rate of change, it's just compounding a lot faster.

We're shortening that time window until we have to actually start worrying about those things. That damage from all those small changes and sometimes big regressions, it will show up later. For the users, it will show up in slowness in the app itself, as you can imagine. For efficiency, no, you will be paying more to run exactly the same workload. You add more logs, you do more things, you still have the same workload. You're just paying more for it. For scalability, you have less and less headroom to grow. As you guys probably can imagine, a scalability problem will become a reliability problem at some point when you cannot absorb that spike anymore, when you cannot grow anymore. The damage will appear later. You need to address it as fast as possible.

Performance Engineering

This is nothing new. We've been developing features and adding features and adding logic our whole careers. In one side, you have people shipping features and shipping things. On the other side, you might have a performance engineering team, or you might have a few perf-minded individuals that care about efficiency and care about latency and making things as efficient as possible, balancing that. You have two opposing forces and keeping things in balance. The problem is that with that increased rate of change, one side of the equation got a lot faster. We're shipping more and perf engineering or folks working in performance problems, they need to adapt. We need to keep up. The answer to that is, of course, we don't want to hire more performance engineers. It's hard to find perf engineers in the first place. Again, not the best answer. We need to make our workflows faster as well.

We need to speed up the cycle on the perf engineering side too. We need to speed up our reactive flows, our reactive loops, where we detect things a lot faster. We can profile that and we can get to the root cause of the problem faster. Not just that. Also, have the agents understand the codebase, propose a solution, implement that thing, and get it out and get benchmarked and get tested. When we actually get a notification, "Great, I already have a fix for that." The benchmarks are looking good again. We can continue. Also, on the active side, we can have a lot of agents in parallel looking for possible optimizations. We can have a lot of different skills working on different types of optimizations. You can be looking at the latency hot path. You can be looking at what methods are actually taking most CPU and try to optimize those. You can be looking at allocations and continuously improve that, or bundle size. They all specialize looking at different things, but working nonstop trying to find those optimizations. You don't have to do that in serial.

We need to adapt. We need to move a lot faster. What happens if we don't adapt as fast as we need? Of course, there'll be a lot of chaos behind the scenes, but beyond that, what's the first few things your users will feel? First things are generally slowness. You add logic, things get slow until it becomes a problem. Your users stop coming. They stop using the app. They cancel their plans, or even worse. Maybe got bad to a point where you're just throwing more errors. You're down for longer periods of time. Things take longer to get fixed, but your users are suffering. Especially for a large consumer-based application like ChatGPT, and it might be the case for where you guys work, you don't want that. That has a deep effect in the business bottom line. Performance affects user retention and acquisition. I modeled that at Netflix.

I modeled that at OpenAI. I'm pretty sure if you guys try to model that as well in your companies, you'll see the correlation between things getting slow and you're throwing more errors and users just canceling plans. Direct impact in the business. One thing I've noticed too, especially for AI applications, is everyone is very focused on inference. Inference is the hard part. Inference is the resources we need to watch all the time. They're really focusing on inference components, like your time to first token or your tokens per second, your throughput, which are important. Your component metrics are super important. Inference is a big chunk of the problem, but it's not the whole journey. It's not the whole problem. We need to measure and keep watching what the users actually feel. When the users come to your product, they're generally trying to solve a problem. They're trying to do something, perform an action, and during that process, while they're trying to perform that action, they have different feelings.

They have different expectations. We need to be able to capture that, and then we need to drill down into. Take, for example, this. This is a bit more focused on a chat app, but the idea is the same. I used the same thing at Netflix. I think Chrome uses the same thing as well. The users have different expectations along the action journey. When he types in a message and clicks enter or clicks submit, he generally expects something to come back really quickly, instant, just feedback to that action, just to know that I'm not hanging here or anything. I got a feedback. I might get a spinner. I might get a thinking thing going on there, but something is going on. It's not on me anymore. The product itself, it's working. I just need to wait a little bit. Then comes something I call first visible value.

When can the user continue doing what he wanted to do in the first place? In our case, if it's a chat request, the user wants to start reading that answer. The time it takes to get the first few tokens, and the user can start reading that. You don't want the users to stop reading because the throughput, the stream cadence is slow. You need to make sure that the stream cadence is pretty good so the user can continue reading without getting blocked and frustrated again. Then, the time it takes to finish that message, the time it takes to complete the task itself. Even that has a lot of nuance. Where do you stop the timer specifically? For a chat response, do you stop at the first token? Do you stop at the first few words? When can a user actually extract some value? It depends on the product.

When you have a thinking response, where do you stop the timer? Do you stop at the thinking tokens, or do you actually stop when the user can do the actual final message? In an image, do you stop the timer when the final image is generated, or the first low-resolution image is generated? There's a lot of nuances, and there's a lot of thinking you need to do around product to actually have those numbers right. Having those intents is super important as a dimension as well because the expectations, they also vary depending on what the user is trying to do. He might be asking a very simple question like, what's the capital of France? I can go to a search engine and get a response pretty quickly, so my expectation is that the response should be pretty quick as well. Great. I need to respond faster on those cases.

I actually fired up an agentic loop that needs to go and check a lot of sources. It needs to do some code edits. It needs to do a lot of things, so I expect that to take a bit longer. The expectation for that is longer as well. Image generation is heavier, the same thing. Users have different expectations.

I have all those different timers, those different wall times, different intents, a lot of different dimensions. Now to be able to act on that, then I go and decompose that into system evidence, into specific components I want to track, things that compose that large wall time. I like to use this chart. There's a lot of different options. My team likes this as well. Your latency breakdown chart or layer cake where you have your distribution on the x-axis and actual latency on the y-axis, and the different layers depending on what's going on. The chart is not the important part here, but there's a lot that happens beyond inference. Of course, this is not to scale numbers or anything. It's just to demonstrate. Yes, there is client work. There is networking time. There is time to fetch data, to do serialization, to do tokenization. There's a lot that goes on beyond just inference.

That breakdown into layers, it helps us actually focus our efforts. Also, the intent I mentioned helps us focus our efforts. If we see that the large chunk of time is not inference itself, it's maybe data fetching, we can focus on that and try to improve that. Having visibility the whole path, it's super important. Otherwise, you just default to inference, which is the first thing that comes to mind to everyone. Inference is just part of the problem, as you can imagine. Take, for example, our chat application. Chat is very simple. The interface is simple on purpose. You go, you type your message. You send that message. You wait for a bit, you get the response back. I don't know if many of you developed this sort of system before, but we can't just forward that message to the inference engine. It does not keep that state of your whole conversation and everything that it needs to properly respond to that.

There's a lot that happens before a message gets sent to the inference engine. There's client work on, of course, assembling everything that needs to be assembled at the client side, fixing things there. Once things come back, we need to verify the user identity. We need to verify the client identity. We need to check the user's plan. We need to check quotas. We need to check the actual user state. There's a lot that needs to happen. Then comes a fairly heavy part of it, which is, we need to fetch context. Conversations, they can get pretty long. We need to fetch all that conversation history. The user might have uploaded files like PDFs, images, whatever, to that conversation. We need to fetch those files as well. If you are in a project, we need to fetch that project context as well. A lot of data that needs to be fetched until we start assembling what actually gets sent to the inference engine.

Great. I have all the data. I have everything I need. There's encoding. There's tokenization, and of course every module is like different tokenization. Everything is tokenized. We need to look at the context window, like, is it exceeding the context window the user has? Maybe we need to do truncation. Maybe we need to do some compaction. Then we need to assemble everything to be sent to the inference engine. There's a lot that goes on even before something gets to the inference engine. Then there's a lot that goes on after, we're streaming the message back, because we need to store the conversation somewhere as well later too. The message, as you can expect, what the user sent, that may be a small string, a small question, is just one small ingredient of that whole equation. Especially for long conversations, they are pretty data heavy. You might have conversations that are megabytes long, tens of megabytes long, maybe hundreds of megabytes long.

There's a lot of requests. There's a lot of data moving around our systems. That data, as you can imagine, it consumes all those resources I mentioned before. We are, of course, consuming a lot of database requests, Blob storage, there's I/O, moving things around. There's a lot of CPU serializing, deserializing things. There is a lot of RAM, because you probably need to hold things in memory for a little bit as well. It consumes all those traditional resources that are not GPU. A message I want to give here is just, when people think about AI, the first thing that comes to mind is GPUs. We need GPUs. GPUs are really important. Don't get me wrong, it's super important. Everything else, all your CPU workloads, they are extremely important as well. There's a lot of resources that are taken to make the whole experience work the way it works, so we need to watch out for those things.

Bringing this up to you, because, as you can imagine, there are code changes, there are improvements, there are shipping rate changes to the actual inference engine, but the rest of the product is a lot larger, as there's a lot more components. There's a lot more code change going to the other components that affect CPU and memory, I/O, all those things, sometimes even more than GPU, so bringing that up. Why is that important? It doesn't matter that I have an extremely fast model, the fastest model, the fastest inference engine, if the rest of the path is slow. The user will get a slow product at the end of the day anyway, and will decide not to use it. We need to focus on the whole experience, the whole path of the request, to make sure things are moving faster. The users get that pretty good feeling when they're using the product.

Human in the Loop

Summing things up a little bit, the product path is getting broader, and it's more than inference. It includes all that data, context, tokenization, streaming. That path is getting heavier and heavier with all the agentic coding, and all the changes we're making, and that's just accelerating. We're consuming more resources, and those resources are not just GPU. Keeping in mind, we need to evolve the way we think about perf engineering. Again, lots of changes need to speed things up on our side as well. We, of course, don't want to ask our engineers across the company to slow down or add roadblocks that make things slower. Every AI PR that is open needs to be reviewed by five humans, so we don't want to add things that will slow down, that will just take away the whole advantage we have of using AI agents. We actually need to speed up the different loops that we use in perf engineering, both the reactive loop when something regresses and we need to fix.

We need that to be faster. We need that to be closer to the actual code, or where the problem originated in the first place, and the active loop itself that needs to be a lot more parallel. Usually, our reactive loop, it takes quite a bit of time. Are there any perf engineers around here that did profiling and optimizing things in the past? The loop is quite slow. You find a regression, maybe from one release to another, you see that your CPU regressed a few percent. What do you do next? You need to do some profiling. You need to go and capture your CPU profiling, which stacks are actually in CPU the most time. Hopefully, you have continuous profiling so you can actually compare a before and after and easily spot where things regressed. If you don't, then things become even a bit more complex. You need to look at the code changes, try to map that to the stacks that are running CPU, and see what regressed so we find an optimization.

Then you need to think of optimizations, and you need to implement those optimizations. Then you need to go and ship it, test it, and see if things fix. All of that is serial, being done by one person, and it just takes a bit too long. The path I'm exploring within a team and the direction I'm seeing things go is having things always on, having our agents always on, working non-stop in both loops, both reactive and also the active loop. We want to automate not only the large regressions, so things you can spot easily from one release to another, where generally you can have an agent triggered automatically by that regression. The agent could go and fire a profile. The agent could actually compare two profiles. We have that today. We have performance skills on Codex to do that. Pretty straightforward if you have all those things working in your systems.

The agent has access to the codebase. He can compare before and after what PRs got into that code change. He can go and propose a fix, and hopefully your systems are safe enough to the point where maybe the agent can deploy that and test if that fix actually had the expected effect. Same goes for drift. Small regressions, we can have fine-grained metrics of maybe code complexity, maybe the volume of network requests that gets executed during a benchmark. We can have small measures to find that drift and act on that drift as well. Of course, the active optimization loop that I mentioned before. The idea here, this is more of an end scenario, that's where we want things to go to. There's a lot of intermediary steps on that. We want everything that happens pre-production, so ideally before a merge, or before something gets deployed, all the boring things be caught by a CI.

Pretty straightforward. We've been doing that for a long time. Nothing new there. As you can imagine, there's a lot of things, especially on the perf side, that don't manifest until they get to production, until they get a specific combination of user workloads. There's no way of catching that pre-production in the first place, so we need to catch that and we need to fix that once we see. Again, active profiling.

A few examples. We can have microbenchmarks running all the time on every single PR. Agents can actually help develop those microbenchmarks. We can find a lot of different things. We can find though that performance, that method regression. We can actually monitor all the methods that are part of our latency critical path, and find specific regressions on those. We can profile memory allocations, and we can see if any specific function is allocating more memory, and we can act upon that. We can watch bundle sizes. Especially if you publish into mobile, you can watch bundle size if these things are increasing and if that exceeds your budget or not. The message here is we've probably been doing that for a long time, but we can automatically act upon it. We can go deeper. We can look at the metrics. We can look at the codebase. We can check all our architecture diagrams and all the information we have about the system running, and the agent can propose a few ideas, a few ways of fixing that.

We can go and implement that. Ideally, you can actually benchmark, run the same benchmarks again, and compare before and after. Yes, many times that fix will not work. It will not fix the problem, and you can just throw that away and continue doing. You can fire multiple threads with multiple ideas. That's the point. Of course, this is end state. Ideally, there are a lot of steps to get to that in the first place. You can actually have a set of skills in Codex that just make that process faster. Instead of manually triggering profiles, collecting things, and analyzing, you can have an agent collect that information and do that comparison for you. It's something that can be easily achieved. The agents are pretty good at chugging a lot of data and comparing things. Pretty easy, you can generate a flame graph out of it if you want.

Fairly straightforward, especially with microbenchmarks where you can reproduce things pretty easily. Same goes for production. You want pretty good observability in production. You want to catch all those different regressions that you see. Again, act upon those. Same thing, you get the same information. The difference here is the safety. Things are a bit trickier if you're asking an agent to deploy something to production and you don't want your users to feel any issues, feel any pain. There's a lot of best practice that you need implemented even before you could do that. You can get almost all the way there. You can actually have an agent submit a PR that implements an experiment of a perf optimization. You can do that. If you're not at a stage where you can fully trust an agent to deploy to production without any supervision, you can get almost all the way there, and it should work really well.

The always-on loop is the one that probably excites me the most. In my team we have a lot of really great perf engineers, and we're in the process of how can I actually translate all that skill set into a set of skills to my agents that can continuously be looking for optimizations in my environment. They could be looking at the code. They could be looking at my traces. They could be looking at logs, whatever, but continuously finding optimizations. It's funny too, the habit people have when they have a very specific niche skill set, they name the agents after themselves. I have a Ben agent. I have a Brendan agent. I have lots of agents named as people. Then you start treating them as people at some point too. It's weird, that always-on loop trying to find opportunities.

Proven Best Practices

A few best practices of things that have been working well for us. When you can get agents to complete full loops, things become a lot more autonomous. You can have a lot of agents doing a lot of things at the same time, if they can complete the loop. Meaning, I start the loop from some event or a continuous loop, I can have an idea. The agent can implement that, and it can do the whole thing all the way through actually measuring if that change you made was positive and it worked. We can continue doing that without any human intervention. Sometimes it's not that easy, and at other times it's a bit easier. If you can focus on trying to build all the safety to allow agents to complete full loops without any intervention, things will become a lot faster. A lot less supervision. A lot less bottleneck on people having to monitor.

Then just have people not clicking accept, accept, accept, or just do YOLO mode and let the agents do their thing. As you can imagine too, the basic things we've been doing for a long time, they become more and more important. Your engineering maturity becomes more important. Having really good test coverage, it's important. Otherwise, how would the agents know what they did was right or it's working? It's functionally working or not? Benchmarks, the same way. How can the agents measure if the improvement they implemented was good or not? Having those benchmarks, having those tests, extremely important, became even more important now. You need your contracts expectations between the different parts of the system. Observability, super important. You need really good coverage about your system so the agent can understand what's going on. If there's blind spots, it's very likely that it might go in the wrong direction because it doesn't have that intuition about the system that the humans might have. That safe rollout and maybe an automated canary analysis or a blue-green push, all those safety measures to get the code out even if maybe there's a bug there, but that shouldn't affect users. Roll out in a safe way.

At the end of the day, all those perf optimizations, that's a search problem. The agents can try the different alternatives. They can try the benchmarks. They can compare things. It's really cool because they can do all of that a lot faster, and they can do all those things in parallel. The agents, they do need a clear signal, again, to know if they're moving in the right direction, if what they did is right or wrong, or if they can discard some work or actually use it. They need all those metrics. If you're actually trying to reproduce a production workload, you need actual reliable workloads to be reproduced by the engine itself. Otherwise, you might be optimizing something that doesn't really match production. Same problem you had before as you are working with perf. Fast feedback is important too. The faster the feedback loop, the more the agent can do without waiting.

If your feedback loop might be 12 hours, then maybe your agent doesn't really add that much value because the human interruption is not that much. If you can have a fast feedback, within a few minutes I can know if I'm in the right direction so I can make a decision and move, in one way or another, things become a lot faster. That measurement you have, it just guides the agent. Having the right measurements is super important. If I'm measuring the wrong thing, the agent will optimize the wrong thing and I'm not actually improving the product in the first place. Defining the metric correctly, making sure you're capturing that. That's why I focus a lot in what the user is perceiving during the whole process. Focus on that. Agents, like I mentioned, they can chug through a lot more information than humans. Of course, you need to manage the context window and what you're actually sending to be useful.

You can send a lot of information, all your observability, all the details about your deploys, all your codebase. If you have actual written down definitions of the architecture of how the system works, it's extremely important. It's all context that the agent uses to define things. Different than humans, you don't have tribal knowledge here. You might have tribal knowledge and memories, but, in general, you want to provide that context so agents can do the right thing. General idea, we are working towards a world where humans are just setting the direction and our agents are just doing all the work, at least the boring work that is not making the decisions and defining what is good or bad.

Key Insights

It's not just growth, growth is increasing, of course, but the development flow changed significantly. We're shipping and making changes a lot faster than we were before. There's arguments around if it's good or bad. We are making a lot more changes than we were before, and that has implications in other areas of development too. Not just performance, but all your developer experience, whoever is supporting your CI engines or your code repositories. There's a lot of impact in different areas to support that rate of change. When you're thinking about AI, the first thing that comes to mind is GPUs. It's not just GPUs. There's a lot of optimizations, a lot of things that happen even before anything gets sent to be processed by a GPU for inference. Last but not least, our perf workflow needs to change. All the different workflows that are affected by that different rate of change, that increased rate of change, it needs to evolve to keep up. Otherwise, our balance will be wrong.

See more presentations with transcripts

Recorded at:

Aug 08, 2026