AI コーディング環境の文脈重視化を提唱

Agentic Development

Augment Code’s Vinay Perneti talks models, harnesses, and context.

Vinay Perneti, VP of Engineering at Augment Code, speaks during a presentation.

Credit:

Augment Code

There are a lot of AI coding applications out there, and as impressive as large language models and the agents they enable have become, many of the most recent developments in AI-assisted development have been in the software that manages those models, not just the models themselves.

Earlier this summer, I spoke with the head of product for Claude Code, Anthropic’s Cat Wu, about that company’s approach to building that software.

Wu repeatedly came back to the same point in our conversation: Anthropic’s models (and those of its direct competitors) are improving so quickly that it makes little sense to plan too far ahead or to build opinionated or limiting features around them. Rather, the Claude Code product team attempts to maintain what they call a lean harness.

A harness is the software built around one or more AI models that determines how they are used. It decides what the models see, what actions they can take, and how they interact with the code base. Think of it like a layer between a model and the developer’s actual project.

It’s not that Claude Code as an application and harness has no opinionated features or design choices, but its product team does seem to err on the side of trusting where the models will take them over the next year.

There are other harnesses besides Claude Code, though. There are OpenAI’s Codex, Google’s Antigravity, open source alternatives like OpenCode, and options from startups like Cursor or Augment Code. Each can have different features, emphasis, or opinions about how a model can or should be utilized in agentic workflows.

One key choice that Claude Code makes is to avoid approaches that build a structured context around a codebase in advance by default.

“Going by the evals, we don’t see a measurable change,” she said of those approaches. “And I think we generally lean more toward shipping a leaner harness with fewer opinionated tools and just letting developers add their own if they want.”

Augment Code has made a substantially different bet, although the two companies are not necessarily testing the same interventions or optimizing for the same outcomes; its product pre-indexes the repository using embeddings, a retrieval model, and a vector database, then retrieves conceptually relevant code. So to get the other side of that discussion, I spoke with Vinay Perneti, Augment Code’s VP of Engineering.

Perneti briefly described Augment Code’s differing approach, offered his own team’s evaluations of the advantages, responded to some of the arguments for a leaner harness, and shared his perspective on some of the concerns developers have about AI tools and agentic workflows more broadly.

A conversation with Vinay Perneti

*This interview has been edited for length and clarity.*

*Ars Technica: Can you explain how Augment Code’s context engine works?*

Vinay Perneti: So, if I take a step back, what is it that a particular developer is trying to achieve? They want a particular task implemented and they give it to an agent. And the interesting thing with agents today is they have a limited context window, and every time they need to essentially go get all of the context needed for that and then work on it.

There’s two approaches to context. One is grep-based. Claude Code and Codex and other agents have done that. The second is the semantic retrieval bit. For Augment, we’ve always taken the semantic route, and I’ll describe the building blocks of that.

I would say there’s two core pieces. One is, we have an embedding and a retrieval model pair that are working in the system, and then you have a vector database and an entire, highly optimized back-end system that makes it possible to retrieve in sub-milliseconds.

*Ars: Does it have an advantage in one kind of code base versus another?*

Perneti: Yeah, it actually turns out the advantage comes in large, private codebases. And here is why I say that.

For all the public, open source repos where most of the benchmarks are run, every single model has basically memorized the repo. These models are large enough that they’re actually able to memorize the whole repo. So when you’re trying to get something done, the models kind of already know where to look so they can get to an outcome quickly.

Whereas when you’re doing this in a private repo, a model has never seen that repo. And now the iteration loop for finding the outcome is much longer, right? If you have a semantic understanding of your entire private repo, you can ask a question, and you get to those outcomes much more quickly.

*Ars: People are concerned about token efficiency. Does this semantic approach help with that?*

Perneti: We’ve certainly seen it in certain situations. In fact, we published a blog post—we ran Terminal-Bench with Claude Code and Augment Code, same model. And we completed at similar accuracy, but we were 33 percent more efficient than Claude Code. So I do see this showing up in terms of making better use of tokens because you’re not spending as much time on the exploration side.

*Ars: Anthropic told me that Claude Code does not see measurable eval gains from including more semantic code-navigation tools. But then I look at your blog, and I see you posting benchmarks and other claims that, actually, this really helps. Are you guys measuring different things when you say this, or what are their evals missing?*

Perneti: Great questions. I don’t know what specific retrieval engine they used, and I think the other mistake that often people conflate is, not all retrieval systems are equal just like not all databases are equal, right? And by that, what I mean is the models and the system that are working together, which is the context engine, makes a big difference in terms of the quality of outcomes as well.

So, for example, at Augment, we spent about 18 months at the beginning of the company being founded in 2022—this is pre-ChatGPT—researching retrieval and embedding models predominantly for large code bases. So there’s a lot of research that has gone into, when you’re trying to achieve a particular outcome, what are the right pieces of code to get in this embedding space? All of that is encoded into our retrieval models.

And doing that in a very, very fast way is the system that we built around it. So when somebody says, ‘Hey, I tried Claude Code with a RAG implementation, but I’m not seeing benefits,’ that’s because the implementation and the context engine are very, very different, if that makes sense.

*Ars: What do you say to the notion that models are improving so fast that building anything with any assumptions at all makes no sense? That’s the argument for a super lean harness or not doing any of this, is just that you don’t know where it’s gonna be in six months or twelve months or whatever as the exponential goes up.*

Perneti: There’s a lot of truth to that, but I think you have to think about it from a couple of different perspectives.

At Augment, the way we’ve been thinking about this is there’s two ingredients that you need for higher quality outcomes, intelligence and context. When models are getting really good, intelligence is going to get exponentially better, no doubt about that. But just because they’re more intelligent does not mean they have the context.

Now, a person can get the context that they want by spending the tokens on it. And that’s where the second dimension comes, which is what all of the engineering leaders are asking right now, is cost. How much of your token budget is going towards context gathering and producing the right outcomes? Are you using the model the right way to get the highest quality outcomes?

And that’s where I think harness design and context matter. So to me, it’s a combo of intelligence and context, and then it’s a systems engineering problem at that point: how do you put the right energy into each of these verticals so that you get the most optimal outcome with the least cost?

*Ars: We have a fair number of readers who are skeptical of AI in software development. I see two objections that are especially common. One of them is that they still feel that they just can’t trust these systems enough, that they’re going to create technical debt that’s not worth it. The other is that these agentic workflows are too expensive. What do you think?*

Perneti: I’ll actually touch on both. Let me preface for both of them. I think the fundamental truth I hope everybody agrees with is models are continuing to improve at an exponential rate. So whatever you’re doing, you want to ride the exponential so that you’re understanding where things are going and you’re orienting yourself to benefit from that.

So, going back to your question around trust and verification and tech debt, it is true if you approach this problem as, ‘I’m handing this off to agents and I’m walking away.’ That’s not how it’s gonna work, and that’s why I point out that it’s teams of humans working with teams of agents, where there are many points where humans are still better suited for judgment, right? Spec reviews is an example. We find that agents are actually not good at writing specs. So you have to really work with an agent to guide the agent to write a good, high-quality spec. But once you have the spec we are at a place, they’re really good at executing. So it makes no sense to not take advantage of that.

The second thing… tech debt is actually very real, by the way. One of the patterns that we noticed internally at Augment—we are obviously very agent-forward—is agents are very good at duplicating code… so we found that we’ve had to do focused sprints on reducing tech debt with agents. The beauty about that is that you can say you can come up with a spec that talks about what it means to reduce tech debt, and they’re really good at executing that. So I think that is the way to tackle that part of it.

And the second part, you talked about cost. I think that there’s an interesting point there. If you believe that this is the way of working moving forward, then how do you get the best outcome for your tokens? And that’s where I think things like context engine make a difference. To me, picking the right model for the right task makes a huge difference.

The second thing—this is much more a philosophical, forward-looking bit—is, as these model capabilities continue to improve in exponential and open source models are going to keep up, there’s going to be a point where I think the proportion of tokens that’s going to frontier labs, to open-source models, is going to start tipping, where your most difficult problem, you’ll still throw at a frontier model, but the coding step that I described—like I know exactly what needs to get done and I know I described that in a spec—an open source model might be able to get that done for you. At that point, your cost drops quite a lot, right? So you need a system that allows you to work this way, and I think the cost will just kind of take care of itself, is how I see it.

*Ars: Thank you. I appreciate you making the time.*

Perneti: Yeah, I really enjoyed the chat, Sam.

Takeaways

Both Wu and Perneti agree that they expect frontier models to continue to improve at or even above the current exponential, at least for the next year. (Neither explicitly addressed a longer time frame.) Both agreed that this is fundamentally changing the way software development is done for many organizations.

The practical dispute, such as there is one, is whether context should be assembled ahead of time or rediscovered during each task—especially with large, private codebases.

They’re also sometimes measuring and optimizing for different things. Wu specifically said that Anthropic did not find “a measurable improvement in performance” from “a few LSPs available.” It’s important to note that what Wu was describing when she said that (an LSP, or language server protocol) is not as broad as what Augment Code’s harness does.

When Perneti and Wu describe evaluations of tools or systems that go beyond grep, they are not always talking about the same thing. That is one possible reason (among others) why they come to different conclusions.

One thing Perneti was able to speak to that Wu did not is the possibility that frontier models like Anthropic’s Opus or Fable may become too expensive to remain the main or only models in developers’ and organizations’ agentic workflows, such that smaller or open-weight models might become more widely used.

As the compute crunch continues, some open-weight models—including those small enough and sufficiently quantized to run on local hardware, or at least on hardware maintained in-house by the organization—now approach relatively recent frontier performance on some coding tasks. It remains possible that they, too, will continue to improve at a rapid rate, such that they become viable for coding agents in many teams and projects. Teams may increasingly reserve expensive frontier models for the hardest problems while routing more routine, well-specified work to these cheaper or locally operated models.

Regardless of which approach is more dominant a year from now, more autonomous workflows do not eliminate the need for engineering judgment. If anything, it may be more valuable, not less, as the increases to the speed and scale of production make bad decisions carry higher costs.

Samuel Axon is the editorial lead for tech and gaming coverage at Ars Technica. He covers physical and generative AI, large language models, software development, gaming, entertainment, and mixed reality. He has been writing about gaming and technology for nearly two decades at Engadget, PC World, Mashable, Vice, Polygon, Wired, and others. He previously ran a marketing and PR agency in the gaming industry, led editorial for the TV network CBS, and worked on social media marketing strategy for Samsung Mobile at the creative agency SPCSHP. He also is an independent software and game developer for iOS, Windows, and other platforms, and he is a graduate of DePaul University, where he studied interactive media and software development.

15 Comments

-

-

-

-

-