トークン億万長者のための極限ハルネスエンジニアリング：100万行コード、日量10億トークン、人間コード・レビューゼロ

We’re proud to release this ahead of Ryan’s keynote at AIE Europe. Hit the bell, get notified when it is live! Attendees: come prepped for Ryan’s AMA with Vibhu after.

Move over, context engineering. Now it’s time for Harness engineering and the age of the token billionaires.

Ryan Lopopolo of OpenAI is leading that charge, recently publishing a lengthy essay on Harness Eng that has become the talk of the town:

fuller discussion between Bret and Ryan

In it, Ryan peeled back the curtains on how the recently announced OpenAI Frontier team have become OpenAI’s top Codex users, running a >1m LOC codebase with 0 human written code and, crucially for the Dark Factory fans, no human REVIEWED code before merge. Ryan is admirably evangelical about this, calling it borderline “negligent” if you aren’t using >1B tokens a day (roughly $2-3k/day in token spend based on market rates and caching assumptions):

search it

Over the past five months, they ran an extreme experiment: building and shipping an internal beta product with zero manually written code. Through the experiment, they adopted a different model of engineering work: when the agent failed, instead of prompting it better or to “try harder,” the team would look at “what capability, context, or structure is missing?”

The result was Symphony, “a ghost library” and reference Elixir implementation (by Alex Kotliarskyi) that sets up a massive system of Codex agents all extensively prompted with the specificity of a proper PRD spec, but without full implementation:

The future starts taking shape as one where coding agents stop being copilots and start becoming real teammates anyone can use and Codex is doubling down on that mission with their Superbowl messaging of “you can just build things”.

Across Codex, internal observability stacks, and the multi-agent orchestration system his team calls Symphony, Ryan has been pushing what happens when you optimize an entire codebase, workflow, and organization around agent legibility instead of human habit.

We sat down with Ryan to dig into how OpenAI’s internal teams actually use Codex, why the real bottleneck in AI-native software development is now human attention rather than tokens, how fast build loops, observability, specs, and skills let agents operate autonomously, why software increasingly needs to be written for the model as much as for the engineer, and how Frontier points toward a future where agents can safely do economically valuable work across the enterprise.

We discuss:

Ryan’s background from Snowflake, Brex, Stripe, and Citadel to OpenAI Frontier Product Exploration, where he works on new product development for deploying agents safely at enterprise scale

The origin of “harness engineering” and the constraint that kicked off the whole experiment: Ryan deliberately refused to write code himself so the agent had to do the job end to end

Building an internal product over five months with zero lines of human-written code, more than a million lines in the repo, and thousands of PRs across multiple Codex model generations

Why early Codex was painfully slow at first, and how the team learned to decompose tasks, build better primitives, and gradually turn the agent into a much faster engineer than any individual human

The obsession with fast build times: why one minute became the upper bound for the inner loop, and how the team repeatedly retooled the build system to keep agents productive

Why humans became the bottleneck, and how Ryan’s team shifted from reviewing code directly to building systems, observability, and context that let agents review, fix, and merge work autonomously

Skills, docs, tests, markdown trackers, and quality scores as ways of encoding engineering taste and non-functional requirements directly into context the agent can use

The shift from predefined scaffolds to reasoning-model-led workflows, where the harness becomes the box and the model chooses how to proceed

Symphony, OpenAI’s internal Elixir-based orchestration layer for spinning up, supervising, reworking, and coordinating large numbers of coding agents across tickets and repos

Why code is increasingly disposable, why worktrees and merge conflicts matter less when agents can resolve them, and what it really means to fully delegate the PR lifecycle

“Ghost libraries”, spec-driven software, and the idea that a coding agent can reproduce complex systems from a high-fidelity specification rather than shared source code

The broader future of Frontier: safely deploying observable, governable agents into enterprises, and building the collaboration, security, and control layers needed for real-world agentic work

Ryan Lopopolo

X: https://x.com/_lopopolo

Linkedin: https://www.linkedin.com/in/ryanlopopolo/

Website: https://hyperbo.la/contact/

Timestamps

00:00:00 Introduction: Harness Engineering and OpenAI Frontier

00:02:20 Ryan’s background and the “no human-written code” experiment

00:08:48 Humans as the bottleneck: systems thinking, observability, and agent workflows

00:12:24 Skills, scaffolds, and encoding engineering taste into context

00:17:17 What humans still do, what agents already own, and why software must be agent-legible

00:24:27 Delegating the PR lifecycle: worktrees, merge conflicts, and non-functional requirements

00:31:57 Spec-driven software, “ghost libraries,” and the path to Symphony

00:35:20 Symphony: orchestrating large numbers of coding agents

00:43:42 Skill distillation, self-improving workflows, and team-wide learning

00:50:04 CLI design, policy layers, and building token-efficient tools for agents

00:59:43 What current models still struggle with: zero-to-one products and gnarly refactors

01:02:05 Frontier’s vision for enterprise AI deployment

01:08:15 Culture, humor, and teaching agents how the company works

01:12:29 Harness vs. training, Codex model progress, and “you can just do things”

01:15:09 Bellevue, hiring, and OpenAI’s expansion beyond San Francisco

Transcript

Ryan Lopopolo: I do think that there is an interesting space to explore here with Codex, the harness, as part of building AI products, right? There’s a ton of momentum around getting the models to be good at coding. We’ve seen big leaps in like the task complexity with each incremental model release where if you can figure out how to collapse a product that you’re trying to.

Build a user journey that you’re trying to solve into code. It’s pretty natural to use the Codex Harness to solve that problem for you. It’s done all the wiring and lets you just communicate in prompts. To let the model cook, you have to step back, right? Like you need to take a systems thinking mindset to things and constantly be asking, where is the Asian making mistakes?

Where am I spending my time? How can I not spend that time going forward? And then build confidence in the automation that I’m putting in place. So I have solved this part of the SDLC.

swyx: [00:01:00] All right.

[00:01:03] Meet Ryan

swyx: We’re in the studio with Ryan from OpenAI. Welcome.

Ryan Lopopolo: Hi,

swyx: Thanks for visiting San Francisco and thanks for spending some time with us.

Ryan Lopopolo: Yeah, thank you. I’m super excited to be here.

swyx: You wrote a blockbuster article on harness engineering. It’s probably going to be the defining piece of this emerging discipline, huh?

Ryan Lopopolo: Thank you. It is it’s been fun to feel like we’ve defined the discourse in some sense.

swyx: Let’s contextualize a little bit, this first podcast you’ve ever done. Yes. And thank you for spending with us. What is, where is this coming from? What team are you in all that jazz?

Ryan Lopopolo: Sure, sure.

Ryan Lopopolo: I work on Frontier Product Exploration, new product development in the space of OpenAI Frontier, which is our enterprise platform for deploying agents safely at scale, with good governance in any business. And. The role of VMI team has been to figure out novel ways to deploy our models into package and products that we can sell as solutions to enterprises.

swyx: And you have a background, I’ll just squeeze it in there. Snowflake, brick, [00:02:00] stripe, citadel.

Ryan Lopopolo: Yes. Yes. Same. Any kind of customer

swyx: entire life. Yes. The exact kind of customer that you want to,

Vibhu: so I’ll say, I was actually, I didn’t expect the background when I looked at your Twitter, I’m seeing the opposite.

Stuff like this. So you’ve got the mindset of like full send AI, coding stuff about slop, like buckling in your laptop on your Waymo’s. Yes. And then I look at your profile, I’m like, oh, you’re just like, you’re in the other end too. Oh, perfect. Makes perfect.

Ryan Lopopolo: I it’s quite fun to be AI maximalist if you’re gonna live that persona.

Open eye is the place to do it. And it’s

swyx: token is what you say.

Ryan Lopopolo: Yeah. Certainly helps that we have no rate limits internally. And I can go, like you said, full send at this stay.

swyx: Yeah. Yeah. So the Frontier, and you’re a special team within O Frontier.

Ryan Lopopolo: We had been given some space to cook, which has been super, super exciting.

[00:02:47] Zero Code Experiment

Ryan Lopopolo: And this is why I started with kind of a out there constraint to not write any of the code myself. I was figuring if we’re trying to make agents that can be deployed into end to enterprises, they should be [00:03:00] able to do all the things that I do. And having worked with these coding models, these coding harnesses over 6, 7, 8 months, I do feel like the models are there enough, the harnesses are there enough where they’re isomorphic to me in capability and the ability to do the job.

So starting with this constraint of I can’t write the code meant that the only way I could do my job was to get the agent to do my job.

Vibhu: And like a, just a bit of background before that. This is basically the article. So what you guys did is five months of working on an internal tool, zero lines of code over a mi, a million lines of code in the total code base.

You say it was cenex, more like it was cenex faster than you would’ve. If you had done it by end. So

Ryan Lopopolo: yeah, that

Vibhu: was the mindset going into this, right?

Ryan Lopopolo: That’s right.

[00:03:46] Model Upgrades Lessons

Ryan Lopopolo: Started with some of the very first versions of Codex CLI, with the Codex Mini model, which was obviously much less capable than the ones we have today.

Which was also a very good constraint, right? Quite a visceral feeling to ask the [00:04:00] model to build you a product feature. And it just not being able to assemble the pieces together.

Which kind of defined one of the mindsets we had for going into this, which is whenever the model just cannot, you always pop open at the task, double click into it, and build smaller building blocks that then you can reassemble into the broader objective.

And it was quite painful to do this. Honestly, the first month and a half was. 10 times slower than I would be. But because we paid that cost, we ended up getting to something much more productive than any one engineer could be because we built the tools, the assembly station for the agent to do the whole thing.

[00:04:43] Model Generations, Build Systems & Background Shells

Ryan Lopopolo: But yeah, so onward to G BT 5, 5, 1, 5, 2, 5, 3, 5 4. To go through all these model generations and see their kind of corks and different working styles also meant we had to adapt the code base to change things up when the model was revved. [00:05:00] One interesting thing here is five two, the Codex harness at the time did not have background shells in it, which means we were able to rely on blocking scripts to perform long horizon work.

But with five, three and background shells, it became less patient, less willing to block. So we had to retool the entire build system to complete in under a minute and. This is not a thing I would expect to be able to do in a code base where people have opinions. But because the only goal was to make the Asian productive over the course of a week, we went from a bespoke make file build to Basil, to turbo to nx and just left it there because builds were fast at that point.

swyx: Interesting. Talk more about Turbo TenX. That’s interesting ‘cause that’s the other direction that other people have been doing.

Ryan Lopopolo: Ultimately I have. Not a lot of experience with actual frontend repo architecture.

swyx: You’re talking that Jessica built the sky. So I’m like, I know the NX team. I know Turbo from Jared [00:06:00] Palmer.

And I’m like, yeah, that’s an interesting comparison.

[00:06:02] One Minute Build Loop

Ryan Lopopolo: The hill we were climbing right, was make it fast.

swyx: Is there a micro front end involved? Is it how how complex react

Ryan Lopopolo: electron base single app sort of thing

swyx: And must be under a minute. That’s an interesting limitation. I’m actually not super familiar with the background shelf stuff.

Probably was talked about in the fight three release.

Ryan Lopopolo: BA basically means that codex is able to spawn commands in the background and then go continue to work while it waits for them to finish. So it can spawn an expensive build and then continue reviewing the code, for example.

swyx: Yeah.

Ryan Lopopolo: And this helps it be more time efficient for the user invoking the harness.

swyx: And I guess and just to really nail this, like what does one minute matter? Like why not five, okay, good. We want no. We

Ryan Lopopolo: want the inner loop to be as fast as possible. Okay. One minute was just a nice round number and we were able to hit it.

swyx: And if it doesn’t complete, it kills it or some something,

Ryan Lopopolo: No.

We just take that as a signal that we need to stop what we’re doing, double click, decompose a build graph a bit to get us to high back under so that we [00:07:00] can able the agent continue to operate.

swyx: It’s almost like you’re, it’s like a ratchet. It’s like you’re forcing build time discipline, because if you don’t, it’ll just grow and grow.

That’s right. And you mentioned that my current, like the software I work on currently is at 12 minutes. It sucks.

Ryan Lopopolo: This has been my experience with platform teams in the past, where you have an envelope of acceptable build times and you let it go up to breach and then you spend two, three weeks to bring it back down to the lower end of the average low bed stop.

But because tokens are so cheap Yeah. And we’re so insanely parallel with the model, we can just constantly be gardening this thing to make sure that we maintain these in variants, which means. There’s way less dispersion in the code and the SDLC, which means we can simplify in a way and rely on a lot more in variance as we write the software.

[00:07:45] Observability, Traces & Local Dev Stack

Vibhu: Lovely.

[00:07:46] Humans Are Bottleneck

Vibhu: You mentioned in your article, like humans became the bottleneck, right? You kicked off as a team of three people. You’re putting out a million line of code, like 1500 prs, basically. What’s the mindset there? So as much as code is disposable, you’re doing a lot of review. A lot [00:08:00] of the article talks about how you wanna rephrase everything is prompting everything, is what the agent can’t see.

It’s kind of garbage, right? You shouldn’t have it in there. So what’s like the high level of how you went about building it, and then how you address okay, humans are just PR review. Like how is human in the loop for this?

Ryan Lopopolo: We’ve moved beyond even the humans reviewing the code as well.

[00:08:19] Human Review, PR Automation & Agent Code Review

Ryan Lopopolo: Most of the human review is post merge at this point.

But post, post merge, that’s not even reviewed. That’s just

swyx: Oh, let’s just make ourselves happy by You

Ryan Lopopolo: haven’t used fundamentally. The model is trivially paralyzable, right? As many GPUs and tokens as I am willing to spend, I can have capacity to work with my hood base.

The only fundamentally scarce thing is the synchronous human attention of my team. There’s only so many hours in the day we have to eat lunch. I would like to sleep, although it’s quite difficult to, stop poking the machine because it makes me want to feed it. You have to step back, right?

Like you need to take a systems thinking mindset to things and [00:09:00] constantly be asking where is the agent making mistakes? Where am I spending my time? How can I not spend that time going forward? And then build confidence in the automation that I’m putting in place. So I have solved this part of the SDLC, and usually what that has looked like is like we started needing to pay very close attention to the code because the agent did not have the right building blocks to produce.

Modular software that decomposed appropriately that was reliable and observable and actually accrued a working front end in these things, right?

[00:09:35] Observability First Setup

Ryan Lopopolo: So in order to not spend all of our time sitting in front of a terminal at most, doing one or two things at a time, invested in giving the model that observability, which is that that graph in the post here.

swyx: Yeah. Let’s walk through this traces and which existed first

Ryan Lopopolo: we started with just the app and the whole rest of it. From vector through to all these login metrics, APIs was, I dunno, half an [00:10:00] afternoon of my time. We have intentionally chosen very high level fast developer tools. There’s a ton of great stuff out there now.

We use me a bunch, which makes it trivial to pull down all these go written Victoria Stack binaries in our local development. Tiny little bit of python glue to spin all these up. And off you go. One neat thing here is we have tried to invert things as much as possible, which is instead of setting up an environment to spawn the coding agent into, instead we spawn the coding agent, like that’s the entry point.

It’s just Codex. And then we give Codex via skills and scripts the ability to boot the stack if it chooses to, and then tell it how to set some end variables. So the app and local Devrel points at this stack that it has chosen to spin up. And this I think is like the fundamental difference between reasoning models and the four ones and four ohs of the past, where these models could not think so you had to put them in [00:11:00] boxes with a predefined set of state transitions.

Whereas here we have the model, the harness be the whole box. And give it a bunch of options for how to proceed with enough context for it to make intelligent choices. So

Vibhu: sales, so like a lot of that is around scaffolding, right? Yes. Previous agents, you would define a scaffold. It would operate in that.

Lube, try again. That’s pivoted off from when we’ve had reasoning models. They’re seeming to perform better when you don’t have a scaffold, right? That’s right.

[00:11:28] Docs Skills Guardrails

Vibhu: And you go into like niches here too, like your SPEC MD and like having a very short agent MG Agent md.

swyx: Yes. Yes.

Vibhu: Yeah. So you even lay out what it is here, but I like

swyx: the table contents.

Vibhu: Yeah.

swyx: Like stuff l