NotionのToken Town：5つの再構築、100以上のツール、MCP対CLI、そしてソフトウェアファクトリーの未来 — NotionのSimon Last & Sarah Sachs

For all those who missed out on London, see you in Miami next week!

Notion, the knowledge work decacorn, has been building AI tooling since before ChatGPT, with many hits from Q&A in 2023 and unified AI in 2024 and Meeting Notes in 2025. At the end of their last Make user conference, Ryan Nystrom teased Notion 3.0’s Custom Agents - and they are finally embracing the Agent Lab playbook!

Sarah Sachs and Simon Last of Notion join us for a deep dive into how Notion built Custom Agents, why it took years and multiple rebuilds to get right, and what it means to turn a productivity tool into an agent-native system of record for enterprise work.

We go inside the product, engineering, evals, pricing, and org design decisions behind one of the most ambitious AI product efforts in software today — from early failed tool-calling experiments in 2022 to agent harnesses, progressive tool disclosure, meeting notes as data capture, and the long-term vision for software factories and agentic work.

We discuss:

Sarah and Simon’s path to launching Notion Custom Agents, and why the feature was rebuilt four or five times before it was ready for production

Why early agent attempts failed: no tool-calling standard, short context windows, unreliable models, and too much complexity exposed to the model

The “Agent Lab” thesis: not just wrapping a model, but understanding how people collaborate and building the right product system around frontier capabilities

How Notion thinks about roadmap timing: not swimming upstream against model limitations, but also building early enough that the product is ready when the models are

Why coding agents feel like the kernel of AGI, and how Notion is thinking about “software factories” made up of agents that spec, code, test, debug, review, and maintain codebases together

How Sarah runs AI engineering at Notion (“notes from Token Town”): objective-setting over idea ownership, low-ego teams comfortable deleting their own work, and a culture designed to swarm around fast-changing opportunities

The “Simon Vortex,” company hackathons, and why security gets pulled in early rather than late

How Notion organizes AI: core AI capabilities and infrastructure, product packaging teams, and a broader company mandate that every product surface must increasingly work for both humans and agents

Why prototypes have become much easier to build internally, and how “demos over memos” changes product development inside a tool the whole company already uses every day

Notion’s eval philosophy: regression tests, launch-quality evals, and “frontier/headroom” evals that intentionally only pass ~30% of the time so the company can see where model capabilities are going

What a “Model Behavior Engineer” is, and why Notion treats eval writing, failure analysis, and model understanding as a distinct function rather than just software engineering

The changing role of software engineers in the age of coding agents, and why the new job looks less like typing code and more like supervising a rigorous outer system of agents, PRs, and verification loops

How the “software factory” should work: specs, self-verification, bug flows, subagents, and minimizing human intervention while preserving the invariants that matter

A live walkthrough of a Notion Custom Agent handling coworking space tenant applications by triaging email, enriching applicants with web search, and writing structured data into a Notion database

How agents compose inside Notion: shared databases as primitives, agents invoking other agents, “manager agents” supervising dozens of specialized agents, and memory implemented simply as pages and databases

Notion’s take on MCP vs CLI: why Simon is bullish on CLI’s self-debugging nature, where MCP still makes sense, and how Sarah thinks about capability, determinism, permissioning, and pricing alignment

The evolution of Notion’s internal agent harness: from early JavaScript coding agents, to custom XML, to Markdown and SQL-like abstractions, to tool definitions, progressive disclosure, and a much shorter system prompt

Why Notion cares about teaching “the top of the class,” building for sophisticated operators rather than abstracting away too much capability for everyone

How agent setup works today: agents that can configure themselves, inspect their own failures, and edit their own instructions — with guardrails around permissions

How Notion prices Custom Agents: credits as an abstraction over tokens, model type, serving tier, web search, and future sandbox costs; why usage-based pricing was necessary; and how “auto” tries to match the right model to the right task

Why Notion is not eager to train a foundation model, where they do fine-tune and optimize today, and why retrieval/ranking is one of the most important investment areas as more searches come from agents rather than humans

Why Meeting Notes became one of Notion’s strongest growth loops: not just as transcription, but as high-signal data capture that powers search, custom agents, follow-up workflows, and the broader system of record for company collaboration

Why Notion is more interested in being the place where collaboration data lives than in building hardware themselves — and how wearables or other capture devices may eventually feed into that system

Sarah Sachs

LinkedIn: https://www.linkedin.com/in/sarahmsachs

X: https://x.com/sarahmsachs

Simon Last

LinkedIn: https://www.linkedin.com/in/simon-last-41404140

X: https://x.com/simonlast

Full Video Episode

Timestamps

00:00:00 Introduction and launching Notion Custom Agents

00:01:17 Why Notion rebuilt agents four or five times

00:03:35 Building for where models are going, not just where they are

00:05:32 The Agent Lab thesis, wrappers, and product intuition

00:08:07 User journeys, leadership, and low-ego AI teams

00:13:16 The Simon Vortex, hackathons, and bringing security in early

00:16:39 Team structure, demos over memos, and building for agents

00:20:25 Evals, Notion’s Last Exam, and the Model Behavior Engineer role

00:27:37 Evals as an agent harness and the changing role of software engineers

00:30:42 The software factory: specs, verification, and agent workflows

00:32:18 Live demo: a custom agent for coworking space applications

00:35:08 Composing agents, manager agents, and memory as pages

00:38:15 Notion Mail, Gmail, native integrations, and tools

00:39:43 MCP vs CLI and the cost of capability

00:44:13 When Notion uses MCP vs building its own integrations

00:47:43 The history of Notion’s agent harness rebuilds

00:55:35 Power users, public tools, and the setup agent

00:58:01 Self-fixing agents, permissions, and “flippy”

01:01:13 Pricing, credits, and choosing the right model automatically

01:09:01 Why Notion isn’t training its own frontier model

01:14:07 Retrieval, ranking, and search built for agents

01:17:27 Meeting Notes as data capture and workflow automation

01:21:18 Wearables, hardware, and Notion as the system of record

01:23:45 Outro

Transcript

[00:00:00] Alessio: Hey everyone. Welcome to the Latent Space podcast. This is Alessio founder of Kernel Labs and I’m joined by swyx, editor of the Latent Space.

[00:00:11] swyx: Hello. Hello. We’re back in the beautiful studio that, uh, Alessio has set up for us with Simon and Sarah from Notion. Welcome.

[00:00:18] Sarah Sachs: Thanks for having us.

[00:00:19] Alessio: Thanks for having us. Yeah.

[00:00:20] swyx: Congrats on the launch recently the custom agents, finally it’s here. How’s it feel?

[00:00:26] Sarah Sachs: We ship things slowly. So it had been in Alpha for a little bit and at the point at which is it’s an alpha, um, there’s a group of people that are making sure it’s ready for prod, and then there’s a group of people working on the next thing.

So sometimes some of these launches are a bit delayed satisfaction, so it’s quite nice to remind yourself all the work you did because we do have a habit of like. Being two or three milestones ahead. Uh, just ‘cause you have to be, you know, you can’t get complacent. Um, but it’s been great that people understood how this is helpful.

And I think that’s just easier in general building AI tools today than it was two, three years ago. People kind of get it and so that user education, um, there’s just, it was our most successful launch in terms of free trials and converting people and things like that. It was really successful, so yeah.

But there’s a lot to build.

[00:01:12] swyx: Making it free for three months helps.

[00:01:16] Sarah Sachs: Yep.

[00:01:17] Simon Last: It was definitely super exciting for me because it’s probably the fourth or fifth time that we rebuilt that.

[00:01:22] swyx: Yes.

[00:01:23] Simon Last: And I mean,

[00:01:24] swyx: you’ve been building this since like 20, 22.

[00:01:26] Simon Last: Yeah, I mean, like, it was even right when we got access to like GPT four in late 20 22, 1 of the first ideas we had is like, oh, okay, let’s make an agent that I, we used the word assistant at the time, there wasn’t really the word, the word agent yet, but, oh, we’ll give an access to all the tools the notion can do, and then it, we run in the background like, like do work for us.

And then we just tried that many times and it just. Was too early. Um,

[00:01:48] swyx: I need to force you to like double click on that. What is too early? What didn’t work?

[00:01:52] Sarah Sachs: We were fine to, like, before function calling came out. We were trying to fine tune with the Frontier Labs and with fireworks, like a function calling model on notion functions.

This is right when I joined. I joined because, um, we needed a manager as Simon was needed to be able to go on vacation. So, uh, that’s, that’s around when I joined, so you can speak much more to it.

[00:02:11] Simon Last: Yeah, we did partnerships with both philanthropic and open AI at different times, uh, to try to, at the time the, I mean, when we first tried, there wasn’t even a constant of like tools yet.

We, we sort of designed our own like, like tool calling framework and then we tried to fine tune the models to, uh, to use it over multiple turns. Um, and because it, it didn’t work well out the box, I think. Yeah. The models are just too dumb and the context thing was also way too short.

[00:02:37] Alsesio: Yeah.

[00:02:37] Simon Last: Um, and yeah, we just kind of banged our head against it for a long time.

Uh, unfortunately it was always like, there was always like sort of. Glimmers that it was working, but um, it never felt quite robust enough to be like a useful, delightful thing. Um, until I would say, uh, the big unlock was probably like Sonic 3.6 or seven, uh, early last year. And that’s when we started working on our agent, which we shipped last year.

Um, and then, and then uh, uh, custom agents, kinda a similar capability and that, that one just took longer because we, we just wanted to get the reliability up a lot higher. ‘cause it’s actually running in the background.

[00:03:14] Sarah Sachs: And the product interface of like permissions and understanding, you know, this custom agent is shared in a Slack channel with X group of people and has access to documents that are surfaced to Y group of people.

And the intersect experts, Y might not be whole. And so how do you build the product around making sure administrators understand that permissioning took multiple swings.

[00:03:35] Alsesio: Everything is hard back at the end of the day. Yeah. I’m curious, like when the models are not working, how do you inform the product roadmap of like, okay, we should probably build, expecting the models to be better at some reasonable pace, but at the same time we need to, you know, you had a lot of customers in 2022.

It’s not like you were a new company or like no user base.

[00:03:54] Simon Last: Yeah, I mean I think there’s always the balance of, you know, like you want to be a GI pilled and thinking ahead and building for where things are going. Uh, but also you wanna be like shipping useful things. And so we always try to like, like keep a balance there.

You know, we. We try to take clear, like a portfolio approach. You know, we’re always working on multiple projects and, and we’re always trying to work on, you know, maintaining things where that have already shipped, like, like shipping new things that are like eminently working well and make them really good.

And, and then we wanna always have a few projects that are a little bit crazy. Um,

[00:04:23] Alsesio: and what are the a GI peel projects that you have today? I’m curious about, uh, you don’t have to share exactly what you’re working on, but I’m curious what are things today that maybe in 18 months people will be like, oh, obviously this was gonna work

[00:04:35] Sarah Sachs: 18 months.

[00:04:37] Alsesio: Yeah, 18 months is, you know,

[00:04:37] Sarah Sachs: it’s a long time and Yeah. Yeah.

[00:04:39] Simon Last: I mean, there’s a number of things happening. I think one thing that’s becoming more clear is I think like, like, uh, coding agents are the kernel of EGI, sort of, everything is a coding agent. Mm-hmm. I think that’s, that’s sort of one, one direction.

Um, and then, yeah, the exciting thing about that is sort of your agent can sort of bootstrap its own software and capabilities and actually debug and maintain them. And so yeah, we’re, we’re, we’re thinking a lot about that. And then, yeah, like, like another category of things that I’m, I’m really excited about is like, uh, we call the software factory also.

People are using this, uh, this, this sort of word. Um, basically it just means can you create sort of like a, as automated as possible, a workflow for developing debugging. Mm-hmm. Merging, reviewing, and maintaining a code base and a service where there’s a bunch of agents working together inside, and like, like how does that work?

[00:05:28] Sarah Sachs: If you think back to your initial question, like, why did this take so long? I think something,

[00:05:32] swyx: I didn’t say that, but Yes. Okay. Go ahead.

[00:05:34] Sarah Sachs: Why, what, what changed over the three and half years of trying

[00:05:37] swyx: it? Exactly. Right. Because most people always say like, it didn’t work yet. Then reasoning models came, then it worked.

I was like, okay, let’s go a little

[00:05:43] Sarah Sachs: bit. That’s, I mean, that’s part of it, but I think the other part of it that I actually think is really what will set notion apart for every new capability is we have like. Two skills that are crucial when it comes to frontier capabilities. One is not letting yourself swim upstream.

So like quickly realizing if you’re just pressing against model capabilities versus not exposing the model to the right information, not having the right infrastructure set up. That and of itself is the skill of intuition. And the second is to see, okay, you’re not swimming upstream. Which direction is the river flowing and what is like, how do we think ahead about the product and start building it even if it’s not great yet, so that when it is there, we’re ready for it.

Right? And like those can sometimes feel like counterintuitive things. Like we can be trying to fine tune a tool calling model when they don’t exist yet. And that the trick is to not do that for too long, but realize that there was something there. And we’ve had a lot of things which like, um, we’re just like not swimming in the right direction with the streams.

I think we had multiple versions of transcription before we got meeting notes, right? Oh, I gotta talk

[00:06:39] swyx: about that. Yeah.

[00:06:40] Sarah Sachs: Yeah. Um, and so. I, I, I think that like we, we really closely partner with the Frontier Labs on capabilities and we also have to have strong conviction on, as those capabilities move.

Notion is about being the best place for you to collaborate and do your work. And how does that narrative change if the way that we work changes?

Yeah.

[00:06:58] swyx: Yeah. You told me you were a fan of the Agent Lab thesis, and this is, this is kind of it, right?

[00:07:02] Sarah Sachs: Right. I show that thesis to so many candidates. Like I have it as like micro chrome autofill.

Um, at this point, like it’s one of my most visitations

[00:07:10] swyx: because like, is this the, here’s why you should work in notion and not open, open eye. I, it’s like,

[00:07:14] Sarah Sachs: here’s, here’s what’s different about it.

[00:07:16] swyx: Yeah.

[00:07:16] Sarah Sachs: And here’s why. It’s not just a rapper. I actually think more and more people understand it’s not just a wrapper.

[00:07:21] swyx: Yeah.

[00:07:22] Sarah Sachs: Um, and by the way, like in the beginning, parts of what we build are wrappers on functionality. That works well, of course, but that’s not really the most, um. I would say that’s not the product that, that drives revenue. And that’s not necessarily always what users need.

[00:07:35] swyx: I mean, you know, notion is the AWS wrapper, but like the, the wrapper is very beautiful and like very, very well polished.

So

[00:07:40] Sarah Sachs: like the analogy,

[00:07:41] swyx: like

[00:07:42] Sarah Sachs: the analogy that I’ve been coming back to his Datadog in AWS

[00:07:45] swyx: Yeah.

[00:07:46] Sarah Sachs: So, uh, Datadog could not exist with, without cloud storage. Right. That it’s kind of fundamental that that works. Um, and AWS has like a CloudWatch product, but Datadog is an expert on understanding how people want observability on the products they launch.

And we’re experts in understanding how people wanna collaborate, and that’s really where our expertise lies.

[00:08:04] swyx: Totally.

[00:08:04] Sarah Sachs: Um, regardless of the tools that we use,

[00:08:07] Alsesio: I’m kind of curious how you think about implicit versus explicit expertise. I feel like Datadog is half and half implicit and explicit. It’s like they understand across markets and industries what engineering teams usually look for.

With notion, it’s almost like more of the expertise is at the edge because you as a platform, you’re like so horizontal that the end user is not really the same. Mm-hmm. Like with Datadog, the end user is always like, yeah, an engineering lead, a kinda like SRE related person with notion. It can be anything.

So I’m curious how you put that expertise into a product versus, you know, obviously it, WS cannot build notion. It’s, that doesn’t quite work in this case, but

[00:08:44] Simon Last: it’s, it’s a little bit differently shaped. I think, you know, a classic vertical SaaS, like the data is kind of like that. They understand their individual customer very deeply.

It’s kinda a narrow slice, um, notion has always been super horizontal. And our, our task has always been to sort of balance these two somewhat opposing forces of like, we’re listening to our customers and what they want us to build. It’s a broad slice. And then also we’re thinking about like, okay, how do we decompose what they want into, uh, nice primitives that are, that are really nice to use and we’ll, we’ll get us like as much bang for the buck as possible.

And then, you know. Maintain the whole system, make it all like, like super clean and nice to use.

[00:09:22] Sarah Sachs: We still have user journeys. I mean, we still focus on like core. I actually think the failure of our team is when we focus too much on what are cools that are, what are tools that are

[00:09:31] Simon Last: mm-hmm.

[00:09:31] Sarah Sachs: Cool tools. I actually think that’s when we make have the least velocity because you still need some sort of focus on a user journey.

So like for instance, we’ll all sit down every Friday and look at the P 99 of like the most token exhaustive custom agent transcript and just look at why it didn’t do well and cut a bunch of tasks. Like we still focus on like, this has, like this should work. Email triaging should work. Mm-hmm.