フロンティア・エコシステムはオープンであるべき：Databricks の Matei Zaharia 氏と Reynold Xin 氏が語る理由

We’re excited to have Databricks join us at AIEWF, among hundreds of the top companies in the AI Engineer ecosystem. LS subscribers can use their discount to get past the late bird pricing and access over $50k in sponsor offers!

Everyone is still talking about Satya’s Frontier Ecosystems post, but few have actually built a (now $175 billion) frontier ecosystem and cloud like our guests today.

From open-sourcing the layer above coding agents to rethinking databases for the agent era, Databricks cofounders Matei Zaharia and Reynold Xin are pushing the company beyond the lakehouse into a full data-and-AI operating system. In this episode, Matei and Reynold join swyx at the 2026 Data + AI Summit to unpack Omnigent, LTAP, Lakebase, agent security, open formats, Mosaic, and why databases may matter more than ever once AI agents start doing real work.

We go deep on Omnigent: Databricks’ open-source meta-harness for combining, controlling, and sharing agents across Claude Code, Codex, Cursor, Pi, custom agents, and internal tools. Matei explains why coding agents and enterprise agents run into the same problems: portability, collaboration, session history, security, spend controls, and the need for a common API above every harness.

Then Reynold walks through Databricks’ database dream: why CDC is brittle enough to joke that it means “continuous data corruption,” why HTAP has been the holy grail of database engineering, and why Databricks thinks LTAP gets most of the benefits by unifying the storage layer instead of collapsing every query engine. We also cover Databricks’ infrastructure scale, the culture behind rapid prototyping, the difference between tech and enterprise customers, Databricks vs Snowflake, whether vector databases should have ever existed, the Mosaic model strategy, Genie, AI Runtime, RL fine-tuning, and the thesis that traditional software gets rewritten once the data is in the right place and agents sit on top.

Databricks began as a company for the big data era. The origination of Spark from the Berkeley AMPLab which eventually turned into the product Lakehouse convinced enterprises that they didn’t need a separate data lake, warehouse, ML platform, and governance layer. They just needed one open foundation where all of their data could live and be reasoned over.

Since then a lot has changed, but data has only become more important. Data is no longer something you keep track of and analyze ad hoc, it’s the necessary context agents need in order to act. So the framing has shifted from “where do we put all of our data?” to “how do we expose the right slice of state, history, permissions, and business logic to an AI system at the exact moment it’s doing work?”

If frontier model performance becomes commoditized, the durable advantage then becomes the company-specific context around them: proprietary data, governed access, operational state, transaction logs, workflows, and feedback loops. Which makes Databricks positioned perfectly.

Now coming fresh off the Data + AI Summit 2026, the company is moving just as fast to keep up, announcing Genie One, Omnigent, LTAP, and many more, indicating a central mission in its newer work: Databricks is trying to become the operating system for enterprise agents.

Models are getting good enough, but agents are only useful if they have the right context, permissions, memory, state, cost controls, and access to live business data. Fundamentally it appears that significantly better model performance in production is a systems problem, one that data guys like us are remarkably well prepared to solve!

We discuss:

Why Databricks built Omnigent as a meta-harness above existing AI agents

Why coding agents and custom enterprise agents need the same infrastructure

The common API for agent sessions, files, streams, tool calls, and cancellation

Why persistent sessions, cloud sandboxes, sharing, search, and collaboration matter

Why Databricks open-sourced Omnigent instead of keeping it proprietary

Databricks’ internal agent usage, cloud sandboxes, and coding workflows

The scale of Databricks: 50–60 million virtual machines a day and exabytes before breakfast

Why agent security needs contextual and stateful policies

How an agent could read confidential docs, install a compromised npm package, and leak data

Why spend control matters when an agent can burn $500 reading logs

Startup opportunities around coding-agent analytics, quality, skills, and spend

LTAP, Lakebase, and why Databricks wants to rethink the database stack

OLTP vs OLAP, CDC, and why data pipelines break at 3 a.m.

Why HTAP has historically been the holy grail of database engineering

Why Databricks thinks LTAP is “HTAP done right”

How writing transactional data into column-oriented formats changes analytics

Why agents need live operational context from databases, not just telemetry

How Databricks prototypes strategic systems without endless process

Enterprise vs tech customers, governance, procurement, and DIY culture

The “second system syndrome” risk of rewriting a database engine

Building a database engine from a decade of traces and quadrillions of data points

Why vector databases should never have been a separate category

Why open formats and AI changed the race with Snowflake

The Mosaic story, DBRX, Genie, document parsing models, and specialized model training

Why model customization and RL fine-tuning may become mainstream

Why “get the data there, slap some agent on top” may rewrite traditional software

Matei Zaharia

LinkedIn: https://www.linkedin.com/in/mateizaharia

X: https://x.com/matei_zaharia

Reynold Xin

LinkedIn: https://www.linkedin.com/in/rxin

X: https://x.com/rxin

Databricks

Website: https://www.databricks.com

X: https://x.com/databricks

Timestamps

00:00:00 Introduction

00:02:22 Omnigent and the Agent Infrastructure Layer

00:08:39 Agent Clouds, Common APIs, and Open Source

00:16:52 Databricks Scale and Internal AI Workflows

00:18:03 Agent Security, Governance, and Spend Controls

00:27:34 LTAP and the Database Dream

00:30:30 CDC, HTAP, and Why Data Pipelines Break

00:34:05 Lakebase, Parquet, and Live Data for Agents

00:36:47 Databricks’ Culture of Fast Prototyping

00:43:40 The Dream Engine and Rewriting the Database Stack

00:51:02 Vector Databases, Query Engines, and LTAP

00:52:36 Databricks vs Snowflake

00:57:48 Mosaic, DBRX, Genie, and Specialized Models

01:03:11 Context, AI Runtime, and RL Fine-Tuning

01:06:15 Why Data + Agents May Rewrite Software

01:07:09 Closing Thoughts

Transcript

Introduction: Databricks, Data + AI Summit, and Founder Dynamics

Swyx [00:00:00]: Matei and Reynold from Databricks, welcome to Latent Space.

Reynold Xin [00:00:06]: Hey, thanks for having us.

Swyx [00:00:07]: Yeah.

Matei Zaharia [00:00:08]: Yeah, thanks so much.

Swyx [00:00:09]: thanks for taking time out. You have your Databricks, Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 people

Reynold Xin [00:00:17]: Yeah, it was

Swyx [00:00:17]: in Berkeley

Reynold Xin [00:00:18]: little meetup at Berkeley, I think

Matei Zaharia [00:00:19]: Yeah

Reynold Xin [00:00:19]: put together

Matei Zaharia [00:00:20]: We were doing these tutorials and, yeah, just teach people Spark.

Swyx [00:00:23]: Yeah. obviously now it’s like, I think like the headline number’s like 100,000 people around the world, 30,000 in person.

Swyx [00:00:30]: it’s a crazy

Matei Zaharia [00:00:31]: Amazing

Swyx [00:00:31]: community. Well, I just saw the keynote.

Swyx [00:00:35]: Ali’s just. Did was it obvious or that back when that Ali would be, like, such a great, like, CEO? Like

Reynold Xin [00:00:42]: Oh

Swyx [00:00:42]: such a great presenter?

Reynold Xin [00:00:43]: What do you think?

Matei Zaharia [00:00:44]: I think among our group of founders it was clear that, I think he’d be the best at this.

Swyx [00:00:50]: Yeah.

Matei Zaharia [00:00:50]: And yeah, it turned out great. And he’s, he’s ramped up on so many topics growing a company. He would just go in and, like, study it and, be talk to all the experts. Like, even if he can’t hire the person, learn enough about, like, finance and sales and whatever it was, and, and go from there. Yeah.

Swyx [00:01:09]: Yeah.

Reynold Xin [00:01:10]: he’s obviously very high IQ and a very high EQ, but it wasn’t. Like, Ali today is quite different from Ali from, like 10 years ago. I think there’s a lot of work that he put in to, get to this point.

Swyx [00:01:20]: Yeah. no, to me the most appealing thing about him is that he’s funny. And like, it, it’s, it’

Matei Zaharia [00:01:26]: It’s true, yeah

Swyx [00:01:26]: it’s hard to make jokes about, data warehouses

Reynold Xin [00:01:30]: About serious topics

Swyx [00:01:31]: security

Matei Zaharia [00:01:32]: Yeah

Swyx [00:01:32]: what have you.

Matei Zaharia [00:01:33]: Oh, yeah. That’s for sure.

Swyx [00:01:34]: Yeah. So you guys launched a whole bunch of things. I’ll, I’ll just name check briefly, the stuff because we’re not gonna cover everything. Omnigentt, your baby. LTAP, your baby, your dream engine.

Swyx [00:01:47]: we’re also gonna cover Genie, cover CustomerLake, you acquired Panther

Matei Zaharia [00:01:52]: Yeah

Swyx [00:01:52]: Open Sharing, and there’s Unity AI Gateway. A lot of these, I think, like, are things that you would expect a Databricks to do. It’s, it’s like part of the roadmap. Everyone in your category has similar things. But I think, probably the two of you are leading the two most unique and differentiated initiatives

Omnigent and the Agent Infrastructure Layer

Swyx [00:02:09]: on, in the landscape. Maybe we’ll start with, Omnigentt we’ll, we’ll, we’ll, we’ll go into it. I do think that a lot of people are exploring this meta harness concept.

Matei Zaharia [00:02:21]: Yeah, totally.

Swyx [00:02:21]: What led you to it?

Matei Zaharia [00:02:22]: Yeah. There were a couple of, like, converging lines, which I think is a good sign that you need something new. So on the one hand, there’s all the coding agent info internally. We have really great, dev infra team. they built something called Isaac, that’s like a wrapper on Claude Code and Codex, and, lets you use them either on the web in, like, sandboxes or, just on your dev machine or on your laptop or whatever. And then, they were adding all kinds of stuff there. And we saw all the more advanced engineers like, were building their own workflows with tons of agents, and they were building their own UIs and stuff on top or even on top of that. And then the other one was, like, us building agents. We ship this, like, data science agent called Genie on the research team, which I lead. We also build a lot of internal ones for various things, and then we have all the customer ones. And all of them running into this thing of like, “Oh, I need to switch model and harness and so on,” every few months. Plus the agent is, like, completely useless if you can’t share sessions with someone and have history and have search and all this, like, layer on top of it for collaboration. I thought a bit about it from both contexts and, at first people thought it was weird. They’re like, “Why are you doing coding agents and custom agents in the same thing?” But I said it’s, it’s the same problems and, you just wanna build the stuff that lets you deliver the agent, maybe control it if you care about security, and, make it portable across things. And then we prototyped some things as experiments. We saw, yeah, we can make it work, and then we built that for real.

Swyx [00:04:06]: I’m wondering if this let’s call it architecture

Matei Zaharia [00:04:11]: Yeah

Swyx [00:04:11]: maps to anything in your careers in the past. like I always think about how a lot of things just tie back to operating systems.

Swyx [00:04:18]: A lot of operating

Matei Zaharia [00:04:19]: Yeah

Swyx [00:04:20]: systems tie back to databases,

Matei Zaharia [00:04:21]: So

Swyx [00:04:21]: or the other way around

Matei Zaharia [00:04:22]: so the thing, I do think it ties a lot to, like, network protocols, internet protocol. we also

Swyx [00:04:29]: Communication between entities.

Matei Zaharia [00:04:30]: Yeah. We did stuff with, like, data sharing also, which is probably, most viewers probably won’t know unless they’

Swyx [00:04:36]: Yeah, open protocol is the term.

Matei Zaharia [00:04:37]: Yeah.

Swyx [00:04:38]: Open sharing. Open sharing.

Matei Zaharia [00:04:38]: Open sharing.

Swyx [00:04:39]: Yes.

Matei Zaharia [00:04:39]: Yeah. So it’s like you have a company, you maintain some table, like let’s say like a Walmart or something. They have like the, inventory and what’s been sold in each store. And then you also have suppliers, and they would love to produce more things and ship them, like, exactly the moment you need them. So they would love, like, real-time access to your table. So instead of like sending emails around or Excel sheets or phone calls, why can’t you share like a view of that table in real time with them? Then they query, they, join it with their data, and they decide what to send. So it’s one of these things where you, like you might ask like today since we can vibe code anything so fast, why do we even need to design like protocols or APIs or software? Why can’t you just vibe code things on demand? But for this type of interoperability where multiple parties that are moving at different speeds are building stuff and you still want some layer on top to coordinate, you do wanna design it and build it. So it reminds me of that, like agents talking to each other and, users talking to agents and tools.

Agent Clouds, Cloud Sandboxes, and Keeping Sessions Alive

Swyx [00:05:42]: Reynold, any other comments alternative viewpoints?

Reynold Xin [00:05:46]: I think, by the way, we had a debate on exactly which set of benefits would, matter a lot, and I think around the time we decided to do this thing I was telling Matei, “Hey,” it just happened to be there’s a particular week that I was coding nonstop

Swyx [00:06:00]: from the moment I woke up to, like, the moment I went to bed, I was, like, looking at my Claude sessions, my Codex sessions. And one of the things that was particularly annoying was having to keep my laptop open.

Swyx [00:06:12]: I was driving to a doctor’s appointment, and I remember because I wanted to make sure the whole thing continues working.

Matei Zaharia [00:06:18]: But by the way, it’s so comforting to hear you say that because I’m like, “I don’t know if I’m a clown and I’m doing this or like.”

Swyx [00:06:25]: Yeah. Like honestly, I was driving and I was tethering my laptop to my phone.

Matei Zaharia [00:06:29]: huh.

Swyx [00:06:29]: Keeping it on the side. Whenever I hit a red light, I started looking at what’s going on my laptop.

Matei Zaharia [00:06:35]: Yeah.

Swyx [00:06:35]: And I just felt that was ridiculous.

Matei Zaharia [00:06:37]: Yeah.

Swyx [00:06:37]: It felt like we went back to the dark ages

Matei Zaharia [00:06:39]: Yeah

Swyx [00:06:40]: programming. the productivity you gain from all this coding age is amazing, but, yeah.

Matei Zaharia [00:06:45]: Have you heard of cloud?

Swyx [00:06:47]: Yeah.

Swyx [00:06:48]: It was crazy to me.

Matei Zaharia [00:06:49]: Oh, the thing you were working on was the sandboxes or was this before that?

Swyx [00:06:52]: It was a sandbox.

Matei Zaharia [00:06:53]: Okay.

Swyx [00:06:54]: I was work

Matei Zaharia [00:06:54]: So you were in

Swyx [00:06:55]: So I was approaching from a very different angle. I wanted to, “Hey, we’re gonna have cloud sandboxes that doesn’t shut down. You can get one very quickly,” but not just for running agentic sessions.

Matei Zaharia [00:07:06]: Yeah.

Swyx [00:07:06]: It’s also for running development. So I was personally building that week, and through building that, I ran into all these issues, and then I wrote

Matei Zaharia [00:07:15]: Yeah

Swyx [00:07:15]: a document for Matei, it’s like, “Here’s my wish list of what the actual environment should do.” And I think he ended up almost implementing

Matei Zaharia [00:07:22]: Yeah

Swyx [00:07:22]: every single one of them.

Matei Zaharia [00:07:23]: Yeah, I remember Reynolds saying, ‘cause my first prototype of this had just chats with your agent and he said, “I have to be able to open a shell, like my own shell and like list files and like tail them and stuff.” So

Swyx [00:07:36]: So SSH into a mainframe.

Matei Zaharia [00:07:37]: Yeah. it has that now.

Swyx [00:07:39]: Tailing my log.

Matei Zaharia [00:07:40]: Yeah.

Matei Zaharia [00:07:41]: Yeah.

Swyx [00:07:41]: And also another thing I think I asked was, I had. I still use cursor for the sole purpose of rendering markdown files.

Matei Zaharia [00:07:48]: huh. Yes.

Swyx [00:07:49]: So I said, “If you just give me a way to see my markdown files and render

Matei Zaharia [00:07:53]: Yeah

Swyx [00:07:53]: them properly, I don’t need a separate tool anymore.”

Matei Zaharia [00:07:55]: Yeah.

Swyx [00:07:56]: And I think you also built that in.

Matei Zaharia [00:07:57]: Yeah, we, yeah, we did that, yeah. Yeah, we had a lot of engineers building, their own vibe coding setup. But then the other thing they all said is like, “Hey, I built something that’s amazing for me, but, like, no one else on the team can use it ‘cause I don’t have a server to collaborate.” And this is why we tried to set up, Omnigent, so you can have a server and have the security, set up in there. So, like log in with Google or whatever and, like securely share stuff. which. And that’s where we’ve seen a lot of other agents like hit things. Like people think they prototyped an awesome agent, but it’s not allowed to connect to like some really important data or whatever because of the security team.

Omnigent Architecture, Open Source, and Common APIs

Swyx [00:08:38]: Yeah.

Matei Zaharia [00:08:38]: So yeah.

Swyx [00:08:39]: Yeah. At this point, so for those watching along on YouTube, we’re gonna putting up a image of the structure here, and we can talk a little bit of the architecture. I think I just want to have people understand, ‘cause like when we’re talking about software, it can be very abstract and like here is what we’re talking about. You’ve worked out in open source this entire platform and there’s a runner component and server component with a uniform API that you’ve, you’ve figured out. any other element and obviously you can plug in all this, persistence layers and compute layers. This is a whole cloud. It’s an agent cloud.

Matei Zaharia [00:09:12]: Yeah. It’s, it’s got these components to work with it. The, a lot of the action happens like on the machine where you deploy your agent too. So whatever you’ve got on there, you can run. But yeah, it’s, I think it’s the minimal thing you want to have hosted, like collaborative agents and to have that server. And one of the reasons we open sourced it is, anyone building agents, this gives them an app they can start with and customize, which we were seeing in Databricks too. Like someone would make a nice, agent app and then other teams would ask, “Oh, can I just use yours for my agent?”

Swyx [00:09:45]: Yeah, I think we had like five or six different agentic frameworks

Matei Zaharia [00:09:48]: Yeah

Swyx [00:09:48]: built by every different team. They do all do more or less the same thing. Yeah, you need to. people wanna take something that works in Forkit, and you might as well have something open source. Yeah, which also was another question, which is interesting for Databricks. Like what do you choose to open source? What do you choose to make it proprietary? It’s in. this goes back to Spark, right?

Matei Zaharia [00:10:05]: Yeah.

Matei Zaharia [00:10:06]: One, so one of the reasons to open source something is if you think it’s a layer that will there’ll be some network effect, it’ll benefit from many, p