AI エージェント体験のためにインフラは進化しなければならない — Modal CTO アクシャト・ブブナ氏に聞く

We’ve been running a bit of an Agent Cloud series surveying all the top inference/compute/cloud providers, from Databricks to Daytona to Railway and, even further back, E2B, but we’re excited to conclude this series returning to Modal, which has just raised a monster $355M Series C.

The cloud was built for developers. But agents are now changing that.

The old infra stack was designed for a human who could read docs, reason through YAML, and understand dashboards to figure out what they need when something broke. While this was painful for developers, it worked since they could fill in missing context in their heads.

However, agents don’t have that luxury. Now in this new era of agents, everything has to be tighter.

Agents need good developer experience too - Modal Blog

They need a place to write code, run it, inspect the output, change the environment, debug failures, and try again. Fast iteration and feedback loops with all the necessary context are crucial for agents to operate properly. Furthermore, sandboxes are a clear representation of this shift as agents can easily spin up isolated environments. This programmatic infra even extends to research:

Two years ago, we were one of the first to cover Modal with CEO Erik Bernhardsson and Alessio designed our favorite LS thumbnail of all time:

At the time, Modal was just a teeny little company with a $17M Series A.

Today, fresh off their $355M Series C, Modal is one of the clearest examples of the agent cloud future being built in real time: a cloud platform moving past traditional web app assumptions toward the workloads AI actually creates such as elastic inference, sandboxes, GPU burst, post-training, background agents, and infrastructure that agents themselves can operate.

In this episode, Modal CTO Akshat Bubna joins swyx and Vibhu to unpack why AI applications don’t fit traditional cloud assumptions, why Kubernetes was never designed for bursty compute-heavy workloads, and why Modal is now shifting from developer experience to agent experience.

We go deep on Modal’s AI infra stack: serverless functions, decorator-based infrastructure, elastic inference for custom models, GPU snapshotting, DeFlash, speculative decoding, Auto Endpoints, sandboxes, persistent storage, networked containers, private IPv6, RDMA, multi-node training, and Modal’s capacity pool across 17 cloud providers. Akshat also explains why RL rollouts can require 100,000 sandboxes, why production agents need hard guardrails, why observability may matter more than reading code, and why AI has made infrastructure exciting again.

We discuss:

Why Kubernetes wasn’t built for bursty AI workloads

How Modal started as a better runtime before becoming an AI cloud

Why Modal added GPUs before ChatGPT

The shift from developer experience to agent experience

Why observability matters when agents are writing the code

Elastic inference for custom models across audio, video, robotics, and comp bio

GPU snapshotting, cold starts, and why inference workloads are so bursty

Why RL rollouts can require 100,000 sandboxes

DeFlash, speculative decoding, and frontier-level inference performance

Auto Endpoints and making optimized inference easier to deploy

What Modal adds beyond vLLM, SGLang, and raw GPU rental

Modal’s 17-cloud capacity pool and supercloud strategy

Networked sandboxes, sidecars, private IPv6, and RDMA

Serverless multi-node training for post-training and research workloads

Auto-research, model-guided sweeps, and agents launching GPU experiments

Compute strategy, capacity planning, and batch tiers

Why production agents need specialized sandboxes and hard guardrails

Modal’s take on managed agents, CI, Gitpod/Ona, Python, TypeScript, and Modal Bench

Akshat Bubna

LinkedIn: https://www.linkedin.com/in/akshat-bubna-188885103

X: https://x.com/akshat_b

Modal

Website: https://modal.com

Timestamps

00:00:00 Introduction

00:00:39 Modal’s origin and why Kubernetes wasn’t enough

00:04:32 Developer Experience → Agent Experience

00:06:21 Modal’s AI cloud primitives

00:09:14 Sandboxes, agent loops, and proto-Cognition

00:12:12 Elastic inference, GPU snapshotting, and 100,000 sandboxes

00:15:24 DeFlash, speculative decoding, and Auto Endpoints

00:19:59 Production-grade inference beyond raw GPUs

00:22:00 Background agents, Ramp Inspect, and the agent lifecycle

00:24:08 Modal’s 17-cloud supercloud strategy

00:26:40 Networked sandboxes, private IPv6, and RDMA

00:32:48 Multi-node training, post-training, and auto research

00:37:36 Compute strategy, capacity planning, and batch tiers

00:40:55 Open models, real-time AI, and production agent infra

00:43:06 Hard guardrails, managed agents, and specialized sandboxes

00:46:06 Why AI made infrastructure exciting again

00:48:30 Model APIs, differentiated products, and agentic video

00:51:50 CI, coding-agent infra, SDKs, and Modal Bench

00:57:28 Closing Thoughts

Transcript

Introduction: Modal, Series C, and the Art Party

Swyx [00:00:00]: We’re here with Akshat, CTO of Modal, together with Vibhu. Congrats on your Series C.

Akshat [00:00:10]: Thank you.

Swyx [00:00:11]: Your party yesterday was amazing.

Akshat [00:00:15]: Yeah.

Swyx [00:00:15]: From all the photos and all the swag.

Akshat [00:00:17]: We had a bunch of art installations, which was fun, seeing, like, our products on pedestals next to, like, Rodin.

Swyx [00:00:25]: Very nice. Very nice. When you started, it was not the GPU inference company. Maybe it was in your mind. Take us back to the origin story.

Modal’s Origin: A New Runtime Beyond Kubernetes

Akshat [00:00:39]: I first met Eric, who’s the CEO, through an investor. Back then Eric was already thinking about building, a new runtime, and he got there thinking through why are workflow orchestration products so hard to use. It’s because you have to run them on Kubernetes. Kubernetes is hard to manage. It’s not built for burstiness and, custom images,

Swyx [00:01:03]: Yeah

Akshat [00:01:03]: It has a terrible developer experience.

Swyx [00:01:05]: And I’ll, I’ll interject

Akshat [00:01:06]: Yeah

Swyx [00:01:07]: For listeners, who are new, we interviewed Eric two years ago, and there’s a bit more of the story there from Spotify and all those things.

Swyx [00:01:14]: And I came across Eric through Data Council because he did that talk on the serverless container stack that you guys did, which was like, that was my first like, “Okay, I need to take Modal very seriously” moment.

Akshat [00:01:26]: Yeah.

Swyx [00:01:26]: But it was still very unclear, like, do I need all this for just my data pipelines?

Akshat [00:01:33]: Yeah. initially what we were thinking about was if we build a better runtime, it’s a very useful primitive in itself. It’s There’s a lot of things that, get solved by serverless functions, like you can do, ETL stuff, you can do job queues, you can do all this, like, bursty processing, which it turns out every company had needs for. but then we also were thinking about this as like, this is a primitive that we can build a whole collection of products on, which are very verticalized. So perhaps data engineering would’ve been the first one, but we were thinking about inference. Back then it was more classical inference, like computer vision stuff and running XGBoosts and whatnot. But we added GPUs to the product a year before ChatGPT came out.

From Serverless Containers to GPU Workloads

Swyx [00:02:19]: Nice.

Akshat [00:02:19]: We just didn’t think it would be that big of a deal.

Swyx [00:02:22]: Yeah, just like add A100.

Vibhu [00:02:23]: Was there any, like, early key problem that really sparked off why you built it?

Akshat [00:02:28]: Yeah. Primarily it’s just, none of the tooling that was out there was built for, one, a really great developer experience, and also there’s a general trend of, a lot of the workloads that we were seeing were very. I wish there was a better word for it, but compute-heavy. Like, they need, one, like, need a lot more resources, so you need to burst up and down a lot, versus like Kubernetes designed for, like, slow scaling and, more for, like, web server use cases. And also there’s just a lot more specialization in, like, what kinds of environments these workloads run in. Like, we had sometimes they need accelerators, sometimes they need different kinds of images, and this is just like a consistent thing that we saw across a lot of companies. That would be the next step.

Software-Defined Infrastructure and Decorator-Based DX

Swyx [00:03:13]: Yeah. Yeah. Be nice. I don’t know how much this factored into the early story, but I wrote a post when I was at Temporal about infrastructure, software-defined infrastructure or something like that.

Akshat [00:03:22]: Yeah, the self-provisioning

Swyx [00:03:23]: Self-provisioning.

Akshat [00:03:24]: Yeah.

Swyx [00:03:24]: Yeah. I can’t even remember my own post.

Swyx [00:03:26]: And then you put me on the landing page.

Akshat [00:03:28]: Yeah. We really like, the term and so we stole it.

Swyx [00:03:32]: Because you had the insight that everything can just be in decorators co-located with the code, right?

Akshat [00:03:37]: Yeah.

Swyx [00:03:37]: Was that a big part of the original

Akshat [00:03:39]: Yes

Swyx [00:03:39]: Story or it was just like a DX layer?

Akshat [00:03:41]: That was, really important because we really didn’t want people to spend, so much time, writing YAML, and it seemed like you could really condense the surface area of what you’re doing, put it in code so you can operate on it just like you operate on other code, and like build stuff that’s more expressive and dynamic. and so yeah, that was always a very important part.

Swyx [00:04:04]: Then the pushback is this is a DSL.

Akshat [00:04:07]: Yeah.

Swyx [00:04:07]: It’s you’re closed source. I am locked into Modal.

Akshat [00:04:11]: Yeah. We never really got pushback for that because the nice thing about Modal is you can bring whatever code you have, and sure, the DSL is at the configuration layer for, what hardware you’re using, how you’re scaling things up, but you still own the code.

Akshat [00:04:27]: And that’s, that’s been an important, part of our story, even as we do inference now.

Swyx [00:04:32]: Yeah.

Vibhu [00:04:32]: How much of do you think still stays the same today? Like if you were to build something today, DevX very important, but I feel like, a lot of this has been changed with just hook it up to an agent, have Claude Code, have Codex implement a tool. there’s very agent native primitives that are different than if I’m doing this myself, right?

Developer Experience → Agent Experience

Akshat [00:04:54]: We’ve changed our SDK team to think about agent experience instead of, developer experience and we think that the same benefits that apply for DX also apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that’s not even typed when it can make a couple of changes in a decorator and it gets this self-provisioning runtime of, being able to see its changes live in action? yeah, it just seems from the customers we talk to, they find Modal is much faster for agents to use versus operating on a different substrate.

Swyx [00:05:34]: Yeah, because like you, again, you co-locate the infrastructure requirements to the code that runs it.

Akshat [00:05:38]: Yeah.

Swyx [00:05:38]: Well, the negative thesis now is that nobody’s looking at their code anymore, so there’s no point.

Akshat [00:05:44]: Yeah, people aren’t looking at code. one thing we still see is really important is observability.

Swyx [00:05:51]: Yeah.

Akshat [00:05:51]: Like how good is your dashboard? And of course, like we have, we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what’s going on and, make judgment calls and whatnot. and that’s I feel like, Maybe more important now than looking at the code itself.

Swyx [00:06:11]: Yes, because like, you can try to treat the code as a black box and then use, see the observable action that comes out of it, and then just prompt a change.

What Modal Is For: AI Cloud Primitives

Akshat [00:06:21]: Yeah.

Swyx [00:06:22]: So I think it takes a bit of restraint to not specialize, to say, “I want to ship a new primitive,” and then just be general purpose.

Swyx [00:06:31]: People ask you, “What are you for?” You’re like, “ I don’t know. We can do this, we can do that.”

Vibhu [00:06:36]: Well, I’d be curious to see, like, okay, if we were to ask you, like, what is Modal for even at a high level? There’s a lot you guys do, sandboxes, GPUs, everything. How do you answer?

Akshat [00:06:46]: Modal is a cloud platform that’s built for, where we’ve built the primitives from scratch for AI applications. and right now it covers, inference, training, batch processing, and sandbox workloads.

Akshat [00:07:00]: But we’re building a lot more

Swyx [00:07:02]: I noticed you didn’t say web server, so there is still a role for, like, the always-on large-scale Kubernetes type things.

Akshat [00:07:09]: Yeah, absolutely. We’re, we’re not trying to compete with the renders of the world, because yeah, we think the differentiator for us is the, are the workloads that need specialized compute, need to scale up and down a lot. yeah, they’re, they’re, they’re just shaped differently.

Working Alongside Frontier Startups

Vibhu [00:07:26]: I think you’re building a lot of it alongside the startups, right? They’re innovating quite a bit, even in your, like, latest blog post. Like, even in the series C, the customers that you mention here, the cognitions, technical ones, ramps and whatnot, they’re, they’re innovating with you, right? And that’s not something AWS is doing directly with.

Akshat [00:07:45]: Yeah, absolutely. I think, this is again classic. We’re a small team. We can move really fast. our engineers are working with our customers and figuring it out. Yeah.

Swyx [00:07:54]: So my first week at Cognition, I walked in, there was someone wearing a Modal shirt. I was like, “What are you doing here?” They’re like, “Yeah, I just. I am embedded inside of Cog.”

Akshat [00:08:05]: Yeah, I think that was Peyton. We sent him over

Swyx [00:08:07]: Yeah.

Akshat [00:08:07]: Because, the latency of communication was too high otherwise.

Swyx [00:08:12]: Yeah, distributed node, you have to - you have to place one and collocate.

Vibhu [00:08:16]: Yeah.

Swyx [00:08:16]: So I had a, I had direct personal experience, right? So I worked on smol developer three years ago. it was inspired by Claude 1. I think you onboarded me at some point, like, just before, and I was like, “Oh, like, I need some bursty compute. Like, I was just gonna try using Modal.” And it was a, it was a pretty pleasant experience. apparently, I showed up in the board meeting, like the analytics.

smol developer, Sandboxes, and Proto-Cognition

Akshat [00:08:39]: Yeah, you blew up on Hacker News and,

Swyx [00:08:41]: Yeah

Akshat [00:08:41]: We got a big traffic spike. I. I think the way you used smol developer was Modal functions for running stuff, which was. Like, the, that was a good use case. but then, yeah.

Swyx [00:08:53]: Yeah. That - So to me, that was proto-cognition.

Akshat [00:08:55]: Right.

Swyx [00:08:56]: If only I had, like, stuck to it.

Swyx [00:08:58]: Like, that was like, if - did you say draw the tech tree

Akshat [00:09:00]: Absolutely

Swyx [00:09:00]: You’re just like, “Yeah, like, probably this will happen.”

Akshat [00:09:02]: Yeah. Like, he was so close. You were just rebuilding upon us

Swyx [00:09:04]: I just didn’t realize.

Akshat [00:09:05]: But the funny story there is at the same time, we were talking to a bunch of customers who needed something like sandboxing.

Swyx [00:09:14]: Yeah.

Akshat [00:09:14]: This is like twenty-three.

Swyx [00:09:15]: Yeah.

Akshat [00:09:16]: So we built

Swyx [00:09:17]: You introduced a new API right after that.

Akshat [00:09:18]: Yeah.

Swyx [00:09:19]: Yes.

Akshat [00:09:19]: Like, we built sandboxes in May of twenty-three before anyone was even knew this was gonna be a thing. And the first example we published was, we took smol developer

Swyx [00:09:28]: Smol developer

Akshat [00:09:28]: And put it in a loop, so the agent can iterate on itself.

Swyx [00:09:33]: Loops are hot these days.

Vibhu [00:09:34]: It’s the looper.

Akshat [00:09:34]: Yeah.

Vibhu [00:09:35]: Loops in. When was this, twenty-three?

Akshat [00:09:38]: Yeah.

Vibhu [00:09:39]: A small check.

Akshat [00:09:39]: Yeah.

Swyx [00:09:39]: It’s like twenty-three. so the. the, those for listeners, like, the problem was the models are not built for any of this, right?

Swyx [00:09:46]: Like, you’re just trying to like. They’re not post-training to understand, like, looping and, like, self-correction and tool calling was there, but, like, also not that great.

Akshat [00:09:55]: Yeah.

Akshat [00:09:55]: I don’t remember if you used tool calling in this one, but yeah, the models would just diverge after like ten iterations and not produce anything meaningful.

Swyx [00:10:03]: Yeah. But like, then. So okay, like now talking to myself three years ago, the answer

Vibhu [00:10:08]: Of course they will get better

Swyx [00:10:09]: Collect all the failures, build benchmark, and then collect all the, examples, build the RL environment

Akshat [00:10:15]: Right

Swyx [00:10:15]: Sell it for like ten billion dollars to Meta.

Swyx [00:10:17]: And then also train a model and then sell that for sixty billion dollars to Elon. And this is

Akshat [00:10:23]: Yeah, of course

Swyx [00:10:23]: The funny machine. Like, it’s like, it’s about the hardware.

Akshat [00:10:28]: It’s hard to have that inherent conviction that the stuff will get that much better.

Swyx [00:10:33]: In retrospect, it’s so fucking obvious.

Akshat [00:10:36]: Fair enough.

Swyx [00:10:37]: Like, what else were we doing back then? I don’t know. anyway. Yeah. So this. That was the start of your sandboxing journey, right? I feel like it didn’t blow up until, like, last year.

Akshat [00:10:49]: Yeah.

Swyx [00:10:50]: So there was like a couple years of quietness.

Akshat [00:10:52]: Exactly, yeah. We were

Vibhu [00:10:53]: I think very underrated product value. Like, my experience with Modal, Charles, before he had joined Modal, met this guy at a hackathon, and he really insisted we wanted to run some small model, not hosted anywhere, and he’s like, “ there’s this cool company, Modal. They’ll like spin up a GPU sandbox, we can throw it on there. They’ll take a Hugging Face link.” And like there’s so much value just right there, right? Like instant hosting, spin it up, spin it down. It’ll stay cold, but we run the demo a few days later, it’ll come back up and like all this stuff in retrospect, like it’s still what we needed like today.

Akshat [00:11:27]: Yeah, it’s still needed today. workload shapes have changed a lot as, we run stuff for people with really massive production scale and, there it’s it’s not about scaling from zero to one, but it’s how do we scale really elastically, from like thousand to fifteen hundred GPUs very quickly in a given region. It’s the same shape problem.

Elastic Inference, GPU Autoscaling, and Custom Models

Vibhu [00:11:50]: Okay. So you look at, say, Cursor Composer, right?

Akshat [00:11:53]: Yeah.

Vibhu [00:11:53]: They had a. “We’ll do RL on a model every couple hours.” you guys have a whole version of RL inference gym and whatnot.

Vibhu [00:12:01]: When you look at workloads like that, you’re doing train runs where you need to scale up, scale down every hour thousands of