動画エージェントモデルが次世代へ — xAI のイーサン・ヘ氏に聞く Grok Imagine の開発秘話（98 分読み）

*We’re announcing AIEWF speakers this week! Take the AI Engineering Survey!*

Today’s guest Ethan first joined us for the LS Paper Club as the lead on NVIDIA Cosmos World Model, but then joined xAI and built Grok Imagine in 3 months:

[Ethan He@EthanHe_42Thrilled to share our new Grok Imagine release 🚀 It is the highest quality, fastest, and most cost-effective video generation model yet. Comes with 720P, video editing and better audio! We listened closely to your feedback and moved fast.

Just six months ago, we had almostxAI @xaiUnderstanding requires imagining. Grok Imagine lets you bring what’s in your brain to life, and now it’s available via the world’s fastest, and most powerful video API: https://t.co/tqQwQVgCEI

Try it out and let your Imagination run wild.2:43 PM · Jan 29, 2026 · 116K Views127 Replies · 107 Reposts · 1.35K Likes](https://x.com/EthanHe_42/status/2016749123198673099)He comes back on Latent Space with some nuclear hot takes: that Video Models primarily get their intelligence from LLMs, not from training on video data, and that the next frontier for truly interactive, realtime, long-horizon world models is to work on LLMs (perhaps Interaction Modelsas well…)

Put it this way: In the near term, the next Sora won’t be a better video model, but a video agent.

Generative Media may more closely follow the evolution of AI coding which went from focusing on one-shot output performance and cost, to multiturn reasoning and planning models for agents and systems that can plan, edit, test, debug, and submit PRs.

At a certain point, coding models got so good that the only significant next step to improve performance was handling the orchestration of these models.

Now as the performance of video models increases significantly across realism, consistency, & prompt adherence while becoming more cost efficient, the next evolution of video generation may also be systems that can plan, generate, edit, critique, and iterate across an entire creative task.

[X Freeze@XFreezeGrok Imagine Agent Mode (Beta) just went live on Grok web

It’s a full creative agent working on one infinite open canvas

Grok Agent plans → generates → edits → iterates everything automatically in the same workspace

Tell it what you want and watch it plan, generate, edit, 2:42 PM · Apr 30, 2026 · 920K Views681 Replies · 1.15K Reposts · 3.97K Likes](https://x.com/XFreeze/status/2049725955208622475)In this episode, Ethan joins swyx and Vibhu to unpack what it actually takes to build frontier image and video systems: data, VAEs, diffusion transformers, audio-video alignment, inference speedups, and the hidden cost of storing and moving massive video datasets. From building NVIDIA’s Cosmos world model to joining xAI as Grok Imagine was being built from zero to one, Ethan He has been at the center of some of the most important work in video generation, multimodal models, and real-time world models.

We go deep on Grok Imagine, how a small xAI team shipped its first multimodal video model in three months, why iteration speed matters more than almost anything in model development, and why many of the biggest gains come from fixing tiny bugs in data and training pipelines.

Video agents are almost a sure bet to be the trend in the coming year. We end with a glance at what’s beyond video agents:

Flipbook caused a minor sensation this year when it was released, but most treat it as a fun demo. Ethan takes it very seriously — with the speed and cost of inference coming down every year, the future of custom video JIT UI is closer than you think. We talked about why videogen models may become the front end of AI, how generative UI could replace traditional HTML/CSS, why world models need to be real-time, interactive, and long-horizon, and why the future of video generation may depend more on language models and agents than on diffusion alone.

We discuss:

• Why fast iteration mattered more than meetings
• Why small training bugs can drive huge model quality gains
• Why coding models may make compute the bottleneck again
• How image and video models are trained with synthetic captions
• The role of VAEs and latent space in frontier video models
• Why image models are the foundation for video models
• The tradeoff between temporal compression and real-time interactivity
• Flipbook, Neural OS, and the future of generative UI
• Why future interfaces may go from user intent to pixels
• The hidden cost of training video models: storage, egress, and GPU hours
• How step distillation and consistency models (like OpenAI sCM) makes video inference orders of magnitude faster
• Grok Imagine 0.9 and large-scale audio-video generation
• Why audio-video alignment is harder than text-video alignment
• Ethan’s definition of world models
• Reference-to-video, video extension, and long-context video generation
• Why xAI’s research communication undersells Grok Imagine
• How xAI culture shaped the speed of development
• AI watermarking, SynthID, and detecting generated media
• Why prompt rewriting matters for video models
• Grok Imagine Agent and the rise of video agents
• Why language models may unlock better video generation
• Robotics, physical AI, and embodied world models
• Why Ethan left xAI and shifted focus toward LLMs
• Self-managed context, memory, and the next frontier for language models

Ethan He

• LinkedIn: https://www.linkedin.com/in/ethanhe42
• X: https://x.com/EthanHe_42

00:00:00 Introduction

00:01:25 From NVIDIA Cosmos to xAI

00:03:24 Building Grok Imagine from Zero to One

00:10:07 How Image and Video Models Are Trained

00:18:53 Video Compression, VAEs, and Real-Time Tradeoffs

00:22:10 Generative UI, Flipbook, and Neural OS

00:32:10 The Cost of Training Large Video Models

00:37:04 Distillation, GANs, and Fast Video Inference

00:41:21 Audio-Video Generation and Grok Imagine 0.9

00:48:34 What Makes a World Model?

00:55:51 Reference Videos, Long Context, and Video Memory

01:00:11 xAI Culture, Research, and First-Principles Building

01:09:45 AI Safety, Watermarking, and Prompt Rewriting

01:13:10 Video Agents and AI-Assisted Creation

01:27:32 Why Language Models Unlock Better Video

01:31:15 Robotics, Physical AI, and Embodied World Models

01:32:38 Why Ethan Left xAI

01:34:16 Self-Managed Context and the Future of LLMs

01:38:43 Ethan’s Career Path and Closing Thoughts

Swyx [00:00:00]: We’re here in the studio with Ethan He, most recently of xAI. Welcome.

Ethan [00:00:10]: Thank you. Glad being here.

Swyx [00:00:11]: We’re also here with Vibhu. you were first coming to us or joining the latent space world because you were working on Kosmos at NVIDIA, and you did a paper. We loved it. you presented it as well, so thank you for doing that.

Ethan [00:00:23]: I’ve actually, I also presented the MoEs twice at latent space.

Swyx [00:00:29]: How did you actually hear about us? Did we reach out to you? Is that how it worked?

Ethan [00:00:33]: No, actually, I-- the community. Like I realized, oh, there is this online community that people talk about AI and also learn from each other through papers every week through the Paperclip. It’s very nice.

Ethan [00:00:49]: I learned a lot.

Swyx [00:00:49]: I think three years stop. We haven’t stopped even on Christmas and New Years. many weeks I want to stop but it keeps going.

Vibhu [00:00:58]: No, that was good. I think you had posted that you worked on a paper, and I was “Oh, very cool. We have Paperclip. Present then.”

Vibhu [00:01:04]: But I might have reached out to you after.

Swyx [00:01:05]: you-- because it’s an amateur club, right?

Swyx [00:01:08]: so it’s very unusual and but we have sometimes paper authors come by and actually explain the paper. Today we just did, the poolside paper, which was apparently very good.

Vibhu [00:01:18]: Came out yesterday.

Vibhu [00:01:19]: pretty interesting, right? Fully open. They talk about everything, systems. So it’s a good one. We’ll, we’ll recommend people to read it.

Swyx [00:01:25]: Bring us up to speed on your transition to xAI, ‘cause I actually don’t even know when you joined. just like tell the, tell the story about the sort of transition.

Ethan [00:01:34]: Before xAI, I was working on Kosmos world model as in-- at NVIDIA. So Kosmos is, it’s a giant video foundation models that can-- that aims to simulate the world and for-- it serves as a foundation of-- for all of the roboticists to build on top of. There, once I built the Kosmos one, I realized as this thing also has a scaling law similar to language model, we need to scale up the video models further. that’s, that’s why I realized I need to move to somewhere with much more compute resources. That’s how I

Swyx [00:02:13]: Than NVIDIA?

Vibhu [00:02:14]: The GPU rich came themselves.

Vibhu [00:02:19]: And timeline-wise, when was Kosmo? It was pretty early, right? It was open world model, open paper, everything.

Ethan [00:02:25]: It was end of twenty-four.

Vibhu [00:02:28]: End of twenty-four.

Ethan [00:02:30]: Then at mid twenty-five, I moved to xAI. At that time-- I joined about the time when xAI was about to build video models and in multi-model models. There were no infra, no data, and no model, and it just-- as a few engineers, we built it in three months and released the first model, Grok Imagine zero point nine.

Ethan [00:02:55]: And since then, I keep working on video models and move more from training and to post-training of the video models. For example, like a reference to videos, kind of like the cameo feature and, video extensions. And, before I left, I worked on a world model, leading a small team to focus on the real-time long horizon video generation.

Swyx [00:03:24]: Can you give like a rough roadmap of okay, you’re on a brand-new team. Grok previously was only text, or they partnered with BFL for their image gen stuff. What do you-- what are the building blocks, right? You have compute, data you can procure somewhere. Like just what are like the sequence of things that people should think about when you’re setting up a new team?

Vibhu [00:03:43]: actually even deeper, not just data you can procure. You guys had to go through getting the data too, right? So you shipped it pretty fast, but yeah

Swyx [00:03:51]: three months is like

Vibhu [00:03:52]: From everything

Swyx [00:03:52]: actually like very surprisingly fast.

Ethan [00:03:55]: One thing I say like thanks to my experience at NVIDIA, ‘cause first time when we were building Kosmos together, we built it, for about a year. So this is like the second time I do it. Roughly have an idea, what to do. I say the most important thing is the talent. Everyone were very strong and clever, very close with each other towards a common goal. So that speed up things a lot. So you reduce the communication bandwidth among people, and everyone can work towards the same goal. It’s, it’s like every day there’s not that much meetings on the calendar, like maybe like a, like a sync a day, and after that it’s, it’s just all building. It was pretty fun at that time.

Ethan [00:04:47]: And another thing is that xAI has very strong foundations of like data inference, model inference, and the supporting there can help the model develop a lot. When I look at, training models, I don’t so actually the top important thing is like how many, how many iterations can you do, per day? and the more iteration can you do, you can, you can train the model much faster. So if you have very strong infra and you have a lot of compute, you can, you can train these models in very short period of time. That can give you a much larger buffer to, for errors, and it also gives you the opportunity to spot more bugs.

Swyx [00:05:46]: What is an iteration? Is it like a few hundred steps or what are you

Ethan [00:05:50]: Let’s say just the train-training the model, like from acquire new data and maybe design new algorithms and train a new model, maybe at smaller scale or

Swyx [00:06:01]: So cycle time for like any hyperparam that you’re searching.

Ethan [00:06:04]: Cycle time and tune to like eval this model. Is this model better than my previous iteration?

Ethan [00:06:11]: So

Swyx [00:06:11]: So it’s like before you, someone had already set this up that you can iterate very quickly.

Ethan [00:06:15]: I think the foundation there is extremely good forDeveloping and research models.

Ethan [00:06:23]: And often I find is it-- this is kind of boring, but like a lot of the improvements does not come from new algorithms. It comes from finding small bugs here and there in the data pipeline, in the, in the model training pipeline. Those give, those give the biggest boost to the model quality.

Vibhu [00:06:46]: It’s interesting, right? So you say it’s like small team, less communication bandwidth, but also a lot of quality is like find little bugs. It seems counterintuitive, right? You have a lot of people, you can iron out more of those, but it’s interesting to see the other side, right?

Swyx [00:07:00]: I also wonder, have you-- do you try using LLMs to look for bugs? I don’t know.

Ethan [00:07:05]: I remember at that time it was mid two thousand and twenty-five, so it’s the coding model wasn’t quite there yet. I remem- I remember like December two thousand and twenty-five, it was extremely good. Yeah, I’ve been, I’ve been using it at that time. It’s, it’s helpful. sometimes it produce codes that are kind of difficult to maintain, even though like the first time it built something extremely fast. But it gave the, like a spaghetti code, thousands of lines that I couldn’t maintain, and the LLM itself couldn’t figure out what’s, what’s wrong and how to improve on top of it. But now I find it much better. Yeah, I want to bring up another point here is now coding models are much more efficient and can help us implement stuff much faster. Compute might become a bottleneck again because previously, like if you want to train a new model, say you want to generate new synthetic data and then or write a new algorithm, it might take a few weeks. And during that period of time, you don’t-- you might not have experiments to run. But now you can build that thing within a few hours, then you can immediately train a model.

Ethan [00:08:24]: Now you have to have enough compute to try all of the ideas. So compute might be the bottleneck of iterating speed again.

Swyx [00:08:36]: yeah, I actually, honestly, I think it’s like kind of a stressful job because you’re “Well, I should be trying everything, and if I’m not, then I’m not doing my job well.”

Vibhu [00:08:48]: there’s also the stress of you’re eating thousands of GPUs per hour, which is very expensive and, compute can go to other researchers.

Swyx [00:08:56]: You got the daddy Elon to

Vibhu [00:08:57]: You got daddy Elon.

Ethan [00:08:59]: It was

Vibhu [00:09:00]: But there’s still finite amount of compute, like you want to use it, you want to use it well, you want more of it.

Ethan [00:09:06]: That was quite stressful indeed. Yeah, I think one thing is the-- with coding models now, like a lot of these jobs can be automated, which is much better. A second, it’s a, it’s a marathon, so you got to maintain good health and, a regular schedule.

Vibhu [00:09:28]: It’s, it’s hard to hear that when you shift from zero to nothing in two months.

Swyx [00:09:32]: and, I think obviously the culture at xAI is very famously, people work very hard. one thing I did want to dive into, in our-- in the notes that you, that you sent ahead of time, you had specific comments about the cost of Video Gen training. presumably this is on the Colossus-1, right? the two hundred megawatt cluster. Any whatever you want to just share on that.

Vibhu [00:09:54]: I think there’s, there’s three things we’re talking about, right? So there’s Video Gen, there’s also the Image Gen model that you put out. Do you want to like complete the, okay, so zero to one, you have a few months. Just what are the stages of create Image Gen model?

Swyx [00:10:06]: Oh, yeah, maybe I got distracted.

Vibhu [00:10:07]: Sorry. and then, from there’s Video Gen, there’s Audio Gen. Would love to get into those next. But what is that first few months like? So small team, a lot of bugs, iterations, but what does it look like? Do we take something off the shelf? Do we just get data compute? What’s, what’s the few months like? How do you go to state-art Image Gen model? How do you just start?

Ethan [00:10:28]: I cannot comment specifically how xAI did, but it’s, it’s a quite standard process. I can draw some, examples from Cosmos. So mainly it’s building a video model, you actually need to build a image model first. And building these two models, the data you need is a hundred percent synthetic pair of language and image or language to video. Because on the, on the internet, actually, the videos don’t naturally associate with text. So you can say, oh, like on YouTube, you have the title and you have the description and the comments

Swyx [00:11:11]: Title

Ethan [00:11:11]: of a video, but usually they’re not relevant to the video itself. And say maybe like the video is a natural scene of mountains or something, and the title is, I’m so happy today.

Ethan [00:11:26]: So they have they have no correlation at all. So the first step is to, you have to generate synthetic pair of language with the videos. So you gather videos from the internet, and you use a VLM to caption the videos. So that part, here’s a question, like how do you, how do you gather VLM to begin with? So if there’s no

Swyx [00:11:55]: You, so you fuse the model, right? Like

Ethan [00:11:57]: Say if there’s no like VLM exists, like how do you generate the text to the beginning, right? It’s, it’s impossible.

Swyx [00:12:04]: I see.

Ethan [00:12:05]: In the beginning, it’s like you ask human to describe the video as detailed as possible.For example, you ask them to describe everything, like all objects, all characters, and all interaction and dialogues in the, in the videos. So that’s in the protocol of Cosmos labeling. We require the objective we give to the labelers was that you have to describe the video as detailed as possible, such that a blind person hears a blob of text can reconstruct what the video is like from their head.

Swyx [00:12:43]: Video or image? You’re talking about images.

Ethan [00:12:44]: Video or image, either one of them.

Vibhu [00:12:47]: This was pretty common when we went from clip and DALL-E, right?

Vibhu [00:12:51]: It’s all training on really detailed captioning of images. So same is applied to video, but instead

Ethan [00:12:57]: same applied

Vibhu [00:12:57]: of using multimodal model to pass in video images and write rich descriptions, you can also

Swyx [00:13:04]: I think there’s this traditional perspective of supervised, or, very highly human curated thing. I feel like there’s a unlock with unsupervised, right? Where like you have enough to bootstrap that you can just throw common corpus on it or, whatever. like unsupervised vision and language pairing, right? Like where you just have, interspersed image and text and it just learns. To me, that is the VLM breakthrough that is different from the clip, different from the LM era.

Ethan [00:13:36]: It’s interesting to see that you kind of need both data.

Ethan [00:13:41]: For example, for the

Swyx [00:13:41]: You need it to bootstrap it up. Yeah

Ethan [00:13:43]: for the generative model training, there’s also usually like a small percentage of unlabeled data. So the model is instructed to generate a video without any text instruction. That can also help the model generalize. So after this stage of generative synthetic pair, so, one important common step is to train a compressor or a tokenizer of the image or videos. So because, if you train-- If you can technically, theoretically train image or video models on pure pixels, but the problem is that the, it’s, it’s a lot of tokens. So like one image, it’s, a thousand by a thousand, it’s like one million tokens, one million pixels. It’s impossible to train transformer on that. So it’s, you need to train a tokenizer, which can go from image to latent space and latent space back to image.

Swyx [00:14:45]: That’s why we named the podcast.

Swyx [00:14:48]: But, basically, you’re talking about vocabulary science.

Ethan [00:14:50]: so vocab.

Swyx [00:14:51]: And so, what is, what is imp-- like a million is impossible?

Ethan [00:14:54]: In generative models, the vocab is continuous. It’s a continuous space. We can think about like you map an image to a vector. It’s a, it’s a fixed length vector. It’s sixteen or forty-eight, something like that. And then you map that vector back to the image space. And the mapping is, has-- The mapping is patch-based. So you say you have

Ethan [00:15:22]: a sixteen by sixteen patch and you match, you map that patch of pixels into this latent space.

Swyx [00:15:29]: We’ve covered this

Vibhu [00:15:30]: This is like the vision transformers

Swyx [00:15:32]: VAEs,

Ethan [00:15:33]: VAEs.

Vibhu [00:15:34]: You basically compress your input, you do your generation, you’re reasoning all that generation in smaller dimension, and then you project back out.

Swyx [00:15:43]: VAE is a form compression, but I think the for me, the patching thing is from VIT, right?

Ethan [00:15:48]: You can make those.

Swyx [00:15:49]: Literally the, yeah, the paper is titled like sixteen by sixteen is all you need. something like that. and then I think also, people make a lot of comparisons with this kind of patching with convolutions.

Swyx [00:16:02]: Which is you’re, you’re kind of re- reconstructing the old paradigm with the new.

Ethan [00:16:05]: Actually, in VAEs, there are, there are both convolution networks and transformers. You can actually do both.

Ethan [00:16:14]: After this VAE, so what you’ve got is you’ve got latent space tokens and you’ve got the language tokens. So now the training of the diffusion transformer, usually generative models use diffusion transformers. It is actually quite standard. It’s, it’s very similar to how you train a language transformer models. It’s not that much difference. It’s just the tokens, the visual tokens in, visual tokens out. The only difference is there’s a denoising process. So you train the model to unmask some of the noise. So you add, you add random noise to the visual tokens, and then you train the model to remove those noise to generate the clean tokens. Any inference, the model can iteratively remove noise from a hundred percent noise.

Swyx [00:17:12]: And then there’s also, to speed things along on the tech tree of diffusion, there’s CFG, and then there’s, there’s also, latent diffusion that, there’s, there’s someone in there. I think, somewhere along the line, obviously, like stability and all these other guys, pioneered a lot of this, architecture. I don’t know if you want to get into that or just, or do the video side up to you.

Ethan [00:17:37]: After you train such model, such image model, the reason it’s a, it’s a foundation for video models is that image models are cheaper to train, and they have much denser connection between language and text. So, sorry, language and images. For example, you train a billion, you train on a billion images, and there’s a mapping from the text to the image. And the cost to train the same, like the, a billion, a billion text to a billion videos, that’s much more expensive because videosNaturally have more tokens than images. Because the diffusion models, their understanding of, language purely come from this mapping. So if you don’t have enough mapping, so if you only train on like a ten million videos or something, there-- you might not see enough language tokens in your training, so your model does not understand human intention enough. So that’s why you really-- you train-- you first train this image diffusion models, and then you bootstrap the video model from there.

Swyx [00:18:53]: One thing I did want to ask, because I-- actually, I think you’re, you’re the first per-- video model person I’ve ever talked to, I think. we’ve, we’ve like talked to Luma and all those folks. There’s all these tricks in video compression where basically frame by frame there’s not that much difference, so actually you don’t have to regenerate or save the whole frame, right? but I think MP4 compression or something else like that.

Swyx [00:19:16]: is it tempting to use that? Or as far as I can tell, everyone just treats it as, “No, we would just generate every frame.” Is that roughly the state-art?

Ethan [00:19:27]: There are a few different approaches. Let’s say first, like you want to just directly use MP4 compression and use that as the tokens for the transformers to train, right? So people actually have tried that, but the main challenge is the latent space for the MP4 tokens were not, were not very comprehensible for the models. It’s, it’s extremely hard to train on that. And there’s a

Ethan [00:20:01]: So that’s why they created VAEs, which creates more continuous, latent space, so the models can understand that latent space and learn from it much easier. Even within the VAEs, there are different difficulties of the latent space. So you can imagine something the simplest, the most naive VAE is like you have an image, and you just shuffle all of the images into a, into a vector. So you don’t need to train any VAEs, right? But that latent space is extremely hard for models to train on top of. That’s why there are some debate on like how do you compress the tokens. So you mentioned like you can compress frame by frame. Also, you can compress, the temporal dimension.

Ethan [00:20:52]: The difference is if you compress the temporal dimension, you get a much higher compression rate. Because there’s temporal redundancy between frames, because, this frame and the last frame, likely they are mostly similar, so there’s only some small difference. for example, I think in 12.1 VAE, they have like a eight by eight by four compression rate. So the four temporal tokens are compressed into one tokens. That can save a lot of, save a lot of the context length. If you do it frame by frame, you have to do maybe like eight by eight by one. Your context length will be four times larger. That being said, the benefit of the frame-- per frame compression, we might come back to this later, is, real-timeness and interactivity. ‘Cause if you, if you strain the output of the model, frame by frame, you can-- the model can respond to any user request immediately. So if you have like a temporal four compression, four times compression, then

Swyx [00:22:06]: It might be laggy

Ethan [00:22:07]: there’s a lag there in nature.

Swyx [00:22:10]: So you’re very pilled on this. let’s just go ahead and bring it up ‘cause we have the visual prepared anyway. There’s some frontier applications of real-time video gen. So Flipbook is one of the examples that went viral recently, right? What is Flipbook?

Ethan [00:22:23]: Flipbook is kind of like a web brow- web browser. You can see like it has the web bro- browser UI on top. The difference is all of the UIs are generated by generative image model in real time, and anything here are fake. But you can, you can explore inside this wor- this imaginary world. Say like we-- here we have engineering the Great Pyramid. Like the model generates this for us to understand how it works, and if we want to navigate around and understand further, we can click on some of the, some of the description here, and the model will generate a new page, new subpage describing the details we want to know about.

Swyx [00:23:14]: So it’s basically kind of we’re playing a video, but it’s pausing for our next interaction, and then it just plays the next thing based on our interaction.

Swyx [00:23:23]: Which is kind of cool.

Vibhu [00:23:25]: and you kind of decide your story. So this was, how do you make a pyramid? levering technique seemed interesting, right? It shows how do you take Okay, I want to know what is this

Swyx [00:23:35]: The demo, the demo tweet had more animation between frames.

Vibhu [00:23:38]: I think it’s just skipping,

Swyx [00:23:39]: Oh, it’s just skipping a lot of frames.

Ethan [00:23:40]: they also have a video mode

Vibhu [00:23:42]: It takes a lot. There’s a lot of people

Ethan [00:23:42]: but, a lot of people are using it.

Ethan [00:23:45]: So it’s not available.

Vibhu [00:23:46]: There’s a live video stream. We can try,

Swyx [00:23:50]: So this is an example of the kind of future that you see at the extreme. We don’t-- we’re obviously not in it today.

Swyx [00:23:56]: But in a world where inference is completely free this is better than generating code and text?

Ethan [00:24:02]: So this is, this is a final state of where Viva will be at for word model, I think. Imagine internet doesn’t exist, and then you type in google.com. Like what should, what should, what should a model show you?the model can imagine something, and this is what the model imagine. And these web pages, they completely do not exist. So I think as the inference costs come down, we are going to have generative UI for everything. If you think about how the coding model works, so they write code for a web page, and they render the code might be con- converted into binary, and the binary render the pixels on the screen. So we in machine learning, every time we have some breakthrough, obviously it’s, it’s more intuit. So why don’t we have like user instruction to the pixel directly? So the generative UI will be user intention to the pixels directly. And say like even if I want email, let’s say everyone have the same interface, but I want, I want it slightly different. I want the email to show to me like a TikTok, so I can swipe left and right for the emails. And or maybe you want something else. We can have completely different things. Or like I have I’m looking at, Instagram stories, and I don’t like the Like button. I always may click it. And, generative UI resolved it. So it’s going to be a revolutionary replacement of the interface. So in the future, we might have much more powerful

Ethan [00:25:50]: LLMs and coding models running behind the scene. And in the, in the front-end, the diffusion model will actually be the front-end to show stuff to you. That’s how I imagine it.

Swyx [00:26:02]: Diffusion front-end, deterministic back-end.

Swyx [00:26:04]: Something like that. I find that very expensive, but,

Vibhu [00:26:08]: I find it interesting you called LLMs writing code on the back end deterministic, but okay.

Swyx [00:26:14]: you write it once

Vibhu [00:26:15]: Compare it to

Swyx [00:26:16]: And then you execute.

Ethan [00:26:17]: If you think about the cost, say, let’s say H100 costs $1 per hour, and if you use this eight hours a day and thirty days, so, every month you’re paying this two forty, you’ll actually not wanna pay for that. That’s even more expensive than Cloud Code Max. But if you think about the compute costs come down like two times every year, and I think the future will likely arrive like within few years.

Vibhu [00:26:49]: It’s everything, right? compute cost comes down, compute gets faster, model gets smarter

Ethan [00:26:54]: More efficient

Vibhu [00:26:54]: model gets smaller.

Swyx [00:26:55]: I don’t know why you say two times, ‘cause I think it’s like 100 times. In language models, it is roughly one hundred to a thousand times every twelve to eighteen months, for the same given level of LMSys, ELO.

Vibhu [00:27:08]: That’s a net of everything, right? That’s model performance alongside compute. So different than just compute costs come down. But, a very interesting future.

Swyx [00:27:19]: So the web designers will have to shout out that accessibility is an issue, right? how do you deal with screen readers or whatever. But yes, this is higher bandwidth storytelling than anything you can possibly generate with code, right? So I think that’s the rough idea.

Ethan [00:27:34]: And I’d like to add a little bit that so human naturally have the maximum bandwidth when we are looking at things, look at videos, and we also have maximum output bandwidth when we are talking. So in the future, it might be something like we talk to AI models, and the AI model responds back with a generative UI. So that would be the maximum input and output bandwidth to interact with AI models before neural link happens.

Vibhu [00:28:06]: And it’s also very custom, right? Some people are very visual, some people are not as visual, right? They prefer the text. But the best thing about generative UI, right, it can also be text.

Swyx [00:28:17]: There’s another project that we wanted to highlight, which is the Neural OS. Kinda similar idea, but here you’re liter