AI の高額化に経営陣が衝撃、戦略の見直し迫られる

KETTLE Like a drug dealer who's hooked you and raised their prices, business leaders are simply shocked to learn the AI their organizations are becoming dependent on is suddenly a lot more expensive. You can listen to the latest episode of The Kettle right here on this page, as well as on Spotify, Apple Music, or YouTube where you can subscribe to get notified of the latest episode. Kettle host Brandon Vigliarolo is joined by Reg reporter Lindsay Clark and contributor Joab Jackson this week to discuss their recent stories about the rising cost of enterprise AI - and one way a popular open source project is trying to fight tokenmaxxing with tokenminning - but the question remains whether such measure will be enough to prevent cost-benefit analyses from popping that bubble. Will the AI industry adapt to the fact it's still unprofitable, blowback from usage-based billing, and a desire to not pay AI models as much as the human devs they're supplementing or replacing? That's what's on the hob for this week's episode. A lightly edited transcript is below: Brandon: Welcome back to The Register Kettle podcast. I'm Reg Reporter Brandon Vigliarolo, and you're going to be absolutely shocked to discover what we're talking about this week. I'm kidding, of course, it's AI. Specifically the fact that it seems the world is starting to wake up to how much it costs to actually run these giant models that are supposed to make life easier for enterprises and their employees, but it seems like they're leading to some invoice shocks. With me to talk about the latest panic over AI costs are Reg reporter Lindsay Clark and our contributor Joab Jackson. Thanks to both of you for coming on. Lindsay Clark: No problem, thank you. Joab Jackson: Thank you. Brandon: So Lindsay, let's start with a story you wrote recently about the fact that C-suite occupants are apparently having trouble getting a handle on new usage-based AI costs. What exactly has them so confused? Lindsay Clark: I get the impression that big companies are diving in with both feet with AI. It's the latest trend. They're using it for a lot of coding and business apps, trying to do stuff in the business with it. KPMG is a massive global consultancy. They provide IT services and outsourcing services. Brandon: And they're the ones who wrote the report, correct? Lindsay Clark: That's right; they wrote the report and they have some skin in the game. They did a survey of more than 2,000 senior execs over 20 countries and found that 29 percent of them struggled to understand the operating costs as they scale with enterprise AI deployments. Nearly half of them were also looking to re-phase their AI deployments when the costs outweigh the expected value. Brandon: Explain re-phase. Are they rethinking the deployment itself or are they changing the scope? What's that mean exactly? Lindsay Clark: It's just slowing down and looking at what they're doing. They're looking at lower-cost models and high-fidelity models. It's looking for a mix of models to deploy rather than just maxing out on the most expensive ones. Brandon: Usage-based billing seems to be a relatively recent development in this space. It was all free samples until we get you in the door to the point where you're dependent on this, and then we realize we actually have to make money off this. Speaking as an AI frontier lab, we're going to have to charge you per token because we're just not able to turn a profit. Lindsay Clark: Anthropic, OpenAI, and GitHub have all moved from a subscription, flat-fee, all-you-can-eat model to usage-based billing based on tokens. The vendors, both the model providers and the application vendors that want you to use AI agents in their applications, want people to jump in with both feet and use this stuff as much as possible. Then, as is typical for the IT sector, they try to change the commercials as we go. Brandon: It's a big enough issue that more than a quarter of C-suite people are getting bill shock and realizing that they might not be able to afford this. But they're maybe a little bit hooked in because their engineers have been using this long enough. I wrote a story recently about an open source tool that someone wrote to test engineers to make sure they're not losing their edge in this environment, because a lot of them are. We've written plenty of stories about developers becoming dependent on this, forgetting how to do some of the basic things they used to be able to do. It gives these AI companies a big inroad to basically say, "Well, now we're going to actually try to make money." But it does put them in a precarious position. If you charge too much, these enterprise customers are going to try to find a way around it, whether it's an open source Chinese model or some other solution, rethinking their deployments and trying to go with smaller, large-scale models. But if you charge too little, you're never going to make enough money. Is this a needle that these AI labs can thread, or is it one that's pointed straight at the bubble? Lindsay Clark: There's a report from Gartner from a few weeks ago that was quite interesting. They had done some research about the cost on this topic for AI-assisted coding. Brandon: Right, this is one you covered back at the end of June, right? Lindsay Clark: That's right. A researcher called Nitish Tyagi was saying that there's a real lack of transparency from the vendors over the costs of their coding agents and they don't have cost optimization tools that you would expect in the cloud, for example. Because of this, the costs of the coding agent per developer was going to exceed the actual salary of the developer in 2028. That is the average salary globally. He was already finding that in areas of the world where salaries are a lot lower, like India, the cost of agents is actually exceeding the salary of the developer. This is because the cost of agents is the same throughout the world, whereas developers get paid differently according to where they're located. Brandon: I imagine the cost of these are maybe never going to exceed the salary of a developer in Silicon Valley; those guys are making a lot of money. Lindsay Clark: Exactly. Yes. Brandon: When I think about that, there's no way that you could have token costs being in the hundreds of thousands of dollars. But the overall global average is still a lot. I can understand that it's probably why you see a lot of companies concerned about the viability of AI deployments, causing companies to rehire the people they're laying off. It's another thing that screams, "How sustainable is this?" Lindsay Clark: That research, to your point about the bubble, means they have to recoup these costs somewhere. On the macro picture, a big investment house was looking at the capex across the industry for AI datacenters, and it was $1.5 trillion over five years until 2030. That's a lot of money, and it has to come from somewhere. Again, on the Gartner research, they were saying that all the model providers all have kind of different ways of doing the billing as well. There's no standard. So if you're looking at a way to approach this the model providers are likely to, then it's very hard. Gartner were encouraging people to take matters into their own hands, to look at optimizing their own usage, minimizing their own usage. And one of the things that came out of that call was that there was no direct relationship between any increase in token consumption and an increase in productivity and coding in this case. And that was very telling. It's not the case that the more that you use, the more that you get out of it. Actually, they were arguing that if you are careful with how you do this and controlled, then not only do you end up consuming less, but the quality of the code you produce is also higher. That was reflected in a conversation I had a few weeks ago with Spencer Kimball, who's CEO of Cockroach Labs. Spencer Kimball is well known in the coding world, spent a long time at Google, and he wrote the GIMP open source image processing tool. He said at Cockroach they don't do tokenmaxxing; they just use a whole bunch of models, including a lot of open source models and the cheapest models. He said there's no point in maxing out a model when you haven't provided the right context because you'll just get more rubbish back. He was in line with what Gartner had said; he's quite circumspect about how you deploy it on the commercial level. Brandon: Speaking of tokenmaxxing, that brings me to the next story. Joab Jackson, you wrote this a little while ago about a Netflix engineer who wrote an open source tool that has become popular very fast. It's apparently saved many users who have adopted it hundreds of thousands of dollars by trimming input to LLMs in order to save token cost. It's the opposite of tokenmaxxing; this is like tokenminning. What exactly did he come up with? Joab Jackson: This was a home project, as all good open source projects started out as. He had gotten a $287 bill from Claude Sonnet for some debugging and MCP type work he was doing. He was curious how you generate that large of a bill from that modest of a workload. He took a look at what he was sending over to Claude, and it's known that most token consumption is from input. You get charged for both token input and output, but most of the bills come from what you're putting into the system. He was inspecting the stuff that his agent was submitting, and the vast majority of it was completely redundant. It wasn't useful instructions; it was database schemas, JSON templates, or a lot of log work. He figured that if he could create a program that would tear out the redundant parts and then submit the useful information, he could save money. Claude does have a number of settings, but you quickly descend into AWS billing hell. There's a cache setting. Every time you give a query to an LLM, you're tagging on your complete history up until that point, so a lot of the same information gets passed up. You can set that history to stay for five minutes or an hour. This is how things get tricky real quick. You can set it for an hour, but it costs twice as much, though you get ninety percent savings in reads. Now you have to do the math of the stuff you're submitting versus your workload. Brandon: So this is basically trimming the fat, where instead of resending all that stuff as token input, it's basically telling the model, "You've already got this, use what you have." Joab Jackson: It's not even that you're being chatty. It's all these needlessly verbose schemas. If you have it execute a Rust command in verbose mode, you're going to get all that verbose stuff, even though the LLM doesn't need it. So he came up with a bunch of little modules that he calls squashers that look at these areas: database compression, JSON trimming, and so on. There are a lot of VC-backed token trimming tools now, but he wanted something inline that worked directly from the command line. Now, just a clarification, this wasn't work he was doing [for Netflix]; he has another job entirely at Netflix, but he built this program on his own. He let a few Netflix engineers try it and they liked it, but it just took off all of a sudden on its own anyway. Brandon: It's not an internal Netflix product; it's his own project. I'm curious how this would work. I'm assuming that this is entirely dependent on a context window? Joab Jackson: This is shaping the context window, basically. Brandon: If we go back beyond a certain point, because context windows have a termination, I'm assuming that at that point you're going to have to start resending stuff if you're still undertaking the same conversation. This is only effective as long as it remains in context. Joab Jackson: I use Gemini quite a bit, and they do keep a cache of the stuff so it does seem like a conversation. But after a certain point, although the conversation is seamless to you, all that data is being loaded back for the LLM to parse once again. Brandon: Gotcha. It's Project Headroom, right? So what's doing the work of clearing that headroom out? Is this another LLM or local algorithms that he's written to do this work? Where is that compression and trimming being performed? Joab Jackson: That's all being done on the client. He isn't using the LLM at all. He used some statistical analysis; say you send over a database table, the LLM probably doesn't need that entire table. It needs the first few entries, the schema, and any outliers to get an understanding, but it doesn't need all 100,000 rows. It's basically a series of tricks that he and other contributors have come up with to tidy it up. Brandon: I think you wrote in the story that his talk mentioned $700,000 in savings for the people who've been using it. You wrote the story at the end of May; do you have any update on how much money this thing has saved and how many people are using it? Joab Jackson: I haven't checked in lately, but originally he had estimated, just from the users who opted into telemetry, that they saved about two hundred billion tokens, which accounts for about $700,000. But there are other users who aren't submitting this information. It's open source, so you can't really track it. I did speak with him fairly recently and he said the project is taking off; it continues to attract attention. Brandon: I didn't see in your story whether there was an estimate of the percentage of a token budget or average query that this was trimming off, or is that going to vary by use case? Joab Jackson: Like sNinety percent of server logs aren't necessary. Seventy percent of JSON can also be cut because a lot of it is formatting. Anything that's routine formatting data you don't really need. He also had a nifty feature where he does some text compression. Everything is reversible, so if the LLM needs to go back and get more information, it can do that. Brandon: So this isn't a permanent thing. It's interesting; we're seeing horror stories about AI costs skyrocketing and huge bills. On the flip side, we see daily limits getting eaten up, leaving users with workflows and projects that are freezing in the middle of a workday. This might be what businesses need to free up tokens and trim AI expenses. What do you guys think? Is this going to be enough, or does something else have to give? Joab Jackson: We talk a lot about how generative AI is evolving, but if you really want to go back to evolution and you want to discuss Darwin, one of the core components was limitation of resources. Everything can grow indefinitely without limitations, but at some point, and that's when the real innovation kicks in, you only have limited resources. We have people complaining about data centers using too much electricity; I think the LLMs now are hitting that point. They're going to have to figure out how to make this work strictly with limited resources. I read a lot of AI research and I'm not seeing a lot of AI efficiency research coming out of the labs. Maybe it's just a necessary next part: we have this technology that we don't fully understand, but we have limits too. We have to start to factor in those limits. Lindsay Clark: I was going to add that as well as people looking at open source tools, the vendors, at least within the database market, see a big opportunity here. I've spoken to and read about a number of database vendors who are trying to improve the efficiency and reduce calls to LLMs. For example, a company called Pinecone which did vector databases a while back before everybody else did, is now looking to create a semantic layer between the agent and the business data and tech environment. You store the basics of the landscape in terms of database schema or how the company does financial processes, so you don't have to make calls to the LLM and create queries to find that out every time afresh. The point is, I think we're going to see a lot of people, perhaps from open source or proprietary vendors, see an opportunity here in selling to companies on the basis that they can reduce the costs of AI agents in their business. Brandon: I would hope that Oracle's working on that because, as we discussed last week, they're one of the most exposed to this bubble. Other big data center players that are courting the AI market, it's Microsoft, it's Amazon, it's Google, they all have big things to fall back on if the bubble ends up popping, whereas Oracle is on the hook for a lot of money without as much going on elsewhere. But if they can turn that database into a semantic layer that sits between and reduces calls, that might be pretty valuable. Lindsay Clark: I don't know, because they're also reliant on OpenAI. Last September, they announced a $450 billion pipeline of committed datacenter spending and the market reacted positively. Then a few weeks later, it turned out OpenAI were on the hook for $300 billion of that. Oracle is borrowing money to build these data centers as well. Joab Jackson: As Microsoft pointed out at the recent Build conference, you're going to need these in-between products anyway because the enterprise is a specialized task. There's domain knowledge you don't want to hand over to the LLM providers. There's going to be a whole set of middleware near the data, for analysis, and that will have to come from the channel. Lindsay Clark: It all depends whether the model builders' revenue forecasts are based on this trend for optimization that we're going to see in the next few years, or are the forecasts based on tokenmaxxing? That could be a big difference. Brandon: Based on what Joab just said, you don't see a lot of efficiency work coming out of these labs. My thought would be it's probably the latter, which probably doesn't bode well for the future of this industry. What do you guys think? Is this another thing that we have to watch out for as a potential part of the bubble? Is this going to exacerbate problems, or is this something that can be conquered and moved beyond without causing industry destabilization? Lindsay Clark: I don't know. We keep thinking about a bubble, there's been lots written about a bubble, but the markets seem fine. There are a lot smarter people than me investing a lot of money in this. But there is also cause for concern in terms of we don't really see exactly what the business model is going to be. Brandon: There aren't a lot of returns yet on investments. Lindsay Clark: Exactly. There are question marks over liabilities. There's a whole other side to this, there's AI agents doing coding, but all the big application vendors are looking to build AI agents that do business work for you in finance and HR, you talk about liability but they're not going to be liable for any of the decisions they make. So there's a question mark over that as well. We've seen bubbles burst in the past. My feeling is the capacity will be used at some point in the future, but there might be a few bumps in the road along the way. Brandon: We still have websites; the dot-com bubble burst but websites didn't go away. AI is not going to go away. It's just its scope and shape and use is probably going to be curtailed or changed. Joab, what are your thoughts? Joab Jackson: Weirdly enough, it reminds me of the dot-com bubble, but also reminds me the tablet craze. Microsoft refactored Windows for the tablet format even though 90% of their users didn't have any sort of touch screen capability. For 18 months, software vendors had to come up with touch-enabled versions and laptop manufacturers had to do touch screens. Eventually we got the iPad and the Surface. The tech industry does tend to go too far in one direction, but eventually it pulls back to gauge customer demand, and a much smaller but more useful industry is there somewhere. Brandon: The big question is whether or not we're entering this phase of constraint-driven Darwinian evolution in the AI industry that will save it